Fixing Kubernetes Evictions: 'Pod ephemeral local storage usage exceeds the total limit'

The Quick Fix

If your Pod is crashing right now, you likely need to raise the ephemeral-storage limit in your deployment manifest. A quick jump from 512Mi to 1Gi or 2Gi usually buys enough breathing room to investigate the underlying cause.

spec:
  containers:
  - name: my-app
    resources:
      limits:
        ephemeral-storage: "2Gi" # Double the limit to stop immediate evictions
      requests:
        ephemeral-storage: "1Gi"

What triggers this error?

Think of ephemeral storage as the temporary workspace on a node's physical disk. Unlike Persistent Volumes, this space is volatile. The Kubelet tracks three specific areas:

Log Files: Everything your app sends to stdout and stderr, typically stored in /var/log/pods.
Writable Layer: Changes made directly to the container's filesystem, like a 200MB temporary file created during a PDF generation.
EmptyDir Volumes: Local data stored in emptyDir mounts that aren't specifically configured to use RAM.

Once the Kubelet notices a Pod crossing its limits.ephemeral-storage boundary, it triggers an Eviction. The Pod is terminated immediately to protect the node's disk from reaching 100% capacity, which could otherwise crash the entire node.

Where is the space going?

1. Chatty Application Logs

If a Java application is stuck in a loop or running in DEBUG mode, it can generate 500MB of logs in minutes. Kubernetes includes these logs in your storage quota until they are rotated.

2. Hidden Caches and Temp Files

Python's pip or Node's npm often download hundreds of megabytes into /root/.cache or /tmp during runtime if you're performing dynamic builds. These files live in the container's writable layer and count against your limit.

3. Uncompressed Image Processing

Processing a 4K video or a batch of high-resolution images can easily swell /tmp usage to several gigabytes, far exceeding a standard 512Mi default.

How to Fix It

Approach 1: Increase Resource Limits

If your workload genuinely requires 2GB of scratch space for an ETL job, update your YAML. Apply the update using kubectl apply -f deployment.yaml to restart the Pods with more headroom.

resources:
  requests:
    ephemeral-storage: "1Gi"
  limits:
    ephemeral-storage: "4Gi" # Provide a generous ceiling for spikes

Approach 2: Use RAM for Temporary Files

For small, high-speed temporary files, use the node's memory instead of the disk. This bypasses the ephemeral storage limit entirely, though it will count toward your Pod's RAM limit.

volumes:
- name: cache-volume
  emptyDir:
    medium: Memory
    sizeLimit: "512Mi"

Approach 3: Offload to Persistent Volumes

Don't use ephemeral storage for heavy-duty tasks like database backups or large ML model downloads. Mount a PersistentVolumeClaim (PVC) instead. Since PVCs have their own dedicated storage, they won't trigger Kubelet evictions even if they fill up.

Approach 4: Tighten Log Verbosity

Reduce disk pressure by changing your app's logging level from DEBUG to INFO. In production, 90% of your logs might be unnecessary noise that is literally killing your Pods.

Verification: Did it work?

Check the current disk usage of your containers using the Metrics Server:

kubectl top pod <pod-name> --containers

To see why a Pod was previously killed, inspect the events. Look for a Reason: Evicted message containing the text "usage exceeds the total limit":

kubectl describe pod <pod-name> | grep -A 3 "Annotations:"

If the fix is successful, the Pod will remain in a Running state and the kubectl get events log will stop showing storage-related warnings.