Fixing AWS EKS Pods Stuck in Pending: 0/3 Nodes Available (Insufficient CPU)

The 2 AM Deployment Failure

You just pushed a critical update to your AWS EKS cluster. The CI/CD pipeline shows a green checkmark, but five minutes later, your application is offline. You run kubectl get pods and see a wall of Pending statuses. Minutes pass, but the containers never start.

When you inspect the pod events, you see this specific bottleneck:

0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

This error indicates that the Kubernetes scheduler checked every node and found zero available CPU capacity. Even worse, it couldn't find any low-priority pods to terminate (preempt) to make room for your new deployment. Your cluster is effectively full.

Step 1: Debugging the "Ghost" Resource Usage

A common point of confusion is seeing a node with low actual CPU usage that still refuses new pods. Kubernetes schedules based on Requests, not real-time consumption. If a node has 2 vCPUs and you have four pods each requesting 450m CPU, that node is 90% full in the eyes of the scheduler—even if those pods are currently idling at 1% usage.

Analyze Pod Resource Requests

Check exactly how much CPU your pending pod is asking for. Look for the Requests section in the output:

kubectl describe pod <pod-name> | grep -A 5 "Requests:"

Audit Node Capacity

Next, see how much CPU is already "reserved" across your nodes. This command identifies the gap between what is physically there and what is logically allocated:

kubectl describe nodes | grep -A 10 "Allocated resources:"

If the "CPU Requests" column shows 90% or higher, the scheduler will reject any pod that doesn't fit into the tiny remaining sliver of capacity.

Step 2: Immediate Fixes

Option A: Right-size Your Pod Requests

Developers often set CPU requests based on guesswork. If your microservice actually uses 50m (0.05 cores) but your manifest requests 1000m (1 full core), you are wasting 95% of your paid resources. Adjust your deployment.yaml to reflect reality:

spec:
  containers:
  - name: api-server
    resources:
      requests:
        cpu: "250m"  # Reduced from 1000m
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"

Apply the update: kubectl apply -f deployment.yaml. If the new request fits the available overhead, Kubernetes will schedule the pod instantly.

Option B: Manual Scaling

If your resource requests are already accurate, you simply need more hardware. You can manually bump the size of your Node Group via the AWS CLI. For example, to increase a cluster from 3 nodes to 5:

aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name <your-eks-asg-name> \
    --desired-capacity 5

New EC2 instances typically take 2 to 3 minutes to join the cluster. Once they show a Ready status, the scheduler will automatically move your pods from Pending to Running.

Step 3: Long-term Prevention

Deploy Karpenter for Just-in-Time Scaling

Manual scaling is a recipe for downtime. Modern EKS setups use Karpenter. Unlike the older Cluster Autoscaler, Karpenter doesn't just add nodes to a group. It looks at your pending pods and provisions the exact EC2 instance type needed—like a c5.large for compute-heavy tasks or an r5.large for memory-hungry ones—and can spin them up in under a minute.

Use the Vertical Pod Autoscaler (VPA)

If you aren't sure what your CPU requests should be, run the VPA in Recommendation mode. It tracks historical usage data and provides a suggested value. This prevents the "Insufficient CPU" error caused by over-provisioning "just in case."

Verification

Confirm the fix by watching the pod lifecycle:

kubectl get pods -w

The status should transition from Pending to ContainerCreating and finally Running. To see exactly where the pod landed, use the -o wide flag:

kubectl get pod <pod-name> -o wide

Key Takeaways

• The Scheduler is a Math Engine: It only cares about requests. It does not look at how much CPU your app is actually using right now.
• Account for EKS Overhead: A t3.medium instance has 2 vCPUs, but EKS reserves about 190m CPU for the system. You only have ~1.81 vCPUs available for your pods.
• Automate or Fail: Running production workloads without Karpenter or Cluster Autoscaler leads to manual intervention and outages.
• Alerting: Set up a Prometheus alert for kube_pod_status_phase{phase="Pending"}. This catches capacity issues before your users do.