Fix Kubernetes OOMKilled: Pod Killed Due to Out of Memory

What OOMKilled Actually Means

Your container tried to use more memory than its configured limit. The Linux kernel's OOM killer stepped in, terminated the process, and Kubernetes reported the exit reason as OOMKilled.

This is not a crash. The application didn't throw an exception or hit a bug — the kernel killed it from the outside. You'll see this in kubectl describe pod:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 20 Mar 2026 10:12:00 +0000
  Finished:     Mon, 20 Mar 2026 10:14:33 +0000

Exit code 137 = 128 + 9 (SIGKILL). That's your tell-tale sign.

Diagnose Before You Touch Anything

Don't just bump the limit and hope for the best. First, figure out how much memory your container actually needs.

Check the current pod status

kubectl describe pod <pod-name> -n <namespace>

Scroll to the Limits and Requests section. Then look for the Last State block — that's where OOMKilled shows up.

Check live memory usage

# Usage across all pods in the namespace
kubectl top pods -n <namespace>

# Per container (useful when a pod has sidecars)
kubectl top pod <pod-name> --containers -n <namespace>

Pull historical memory metrics

If you have Prometheus, query the working set to see peak usage over time:

container_memory_working_set_bytes{pod="<pod-name>", container="<container-name>"}

Say your container peaks at 420Mi but your limit is 256Mi. That's a 164Mi gap. Now you have a number to work with, not a guess.

Fix 1: Increase the Memory Limit

Most of the time, this is the fix. Edit your deployment and raise limits.memory:

kubectl edit deployment <deployment-name> -n <namespace>

Or update your manifest directly:

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"   # was 256Mi, doubled it
    cpu: "500m"

kubectl apply -f deployment.yaml

Rule of thumb: set the limit to 1.5–2x the observed peak working set. If your app peaked at 420Mi, try 640Mi. Don't jump to 4Gi — that just masks the real problem until your node runs out of memory.

Fix 2: Drop the Limit Entirely (Know the Risk)

Some workloads have genuinely spiky memory — batch jobs, ML inference, video processing. For those, keeping only a request and no limit can make sense:

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  # no limits block

Without a limit, the container can use whatever's free on the node. The danger is real though: a memory leak in one pod can starve every other pod on that node. Only go this route if your node capacity planning is solid and you're actively monitoring usage.

Fix 3: Find and Fix the Memory Leak

Raised the limit twice and still getting OOMKilled? Your app is leaking. Start with the logs from the container that was killed:

# Logs from the previous (killed) instance
kubectl logs <pod-name> -n <namespace> --previous

Where to look depends on the runtime:

• Java/JVM: forgetting -Xmx means the JVM grows until the kernel kills it. Set heap size explicitly: -Xms256m -Xmx384m. Also cap metaspace: -XX:MaxMetaspaceSize=128m.
• Node.js: the default V8 heap cap is ~1.5GB on 64-bit — way too high for most containers. Add --max-old-space-size=384 to your start command.
• Python: large DataFrames, unbounded caches, or circular references that the GC can't collect. Check with tracemalloc or memory-profiler.

JVM users hit a classic trap: they set the container limit to 512Mi and -Xmx to 512Mi, then wonder why the container gets killed. The JVM needs memory beyond the heap — metaspace, thread stacks, GC overhead — often 100–150Mi extra. If your heap is 512Mi, your container limit should be at least 650–700Mi.

# Safe JVM settings for a 512Mi container
ENV JAVA_OPTS="-Xms128m -Xmx384m -XX:MaxMetaspaceSize=128m"

Fix 4: Let VPA Tell You the Right Numbers

Not sure what limit to set? Run VPA in recommendation mode. It watches real usage and suggests values — without auto-applying anything:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"   # recommendations only, won't restart pods

kubectl apply -f vpa.yaml

# Check back after an hour or two
kubectl describe vpa my-app-vpa -n <namespace>

VPA will show recommended requests and limits based on what it observed. Use those as your baseline.

Verify the Fix

Watch the rollout:

kubectl rollout status deployment/<deployment-name> -n <namespace>

On the new pod, confirm OOMKilled is gone:

kubectl describe pod <new-pod-name> -n <namespace> | grep -A5 "Last State"

Empty Last State or a clean exit means you're in good shape. Then watch memory over the next hour to make sure it's not creeping up:

watch kubectl top pods -n <namespace>

Prevention

• Always set both requests and limits. Pods without limits can consume all node memory and trigger cascading OOMKills across workloads that had nothing to do with the problem.
• Alert before the kill happens. In Prometheus, fire an alert when working set exceeds 80% of the limit for more than 5 minutes. That gives you a window to act.
• Use LimitRange to catch teams that forget:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
  - default:
      memory: 256Mi
    defaultRequest:
      memory: 128Mi
    type: Container

• Load test before you deploy. Run your app under realistic traffic and capture the memory peak. Set limits from measured data, not gut feeling.
• Watch the trend, not just the peak. Memory that spikes and comes back down is normal. Memory that climbs steadily over 6–12 hours without leveling off is a leak.