Fixing 'no IP addresses available' in Kubernetes CNI

The 2 AM Pager Call

It starts with a failed deployment. You notice your pods are stuck in ContainerCreating for ten minutes. When you run kubectl describe pod, you see the dreaded message:

Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Warning  FailedCreatePodSandBox  2m (x15 over 12m)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container network: no IP addresses available

Your CNI (Container Network Interface) has run out of IP addresses. It cannot assign a network identity to new pods. This effectively paralyzes your cluster, whether you're running on AWS EKS, Azure AKS, or a bare-metal Calico setup.

TL;DR: Quick Fixes

AWS VPC CNI: Your subnets are likely full. Check the VPC Console. You might need to add a secondary CIDR or lower WARM_IP_TARGET if nodes are hoarding IPs.
Calico: Your IPPool is maxed out. Add a new, larger IPPool block immediately.
Azure CNI: You've hit the pre-allocated IP limit per node. You often have to redeploy the node pool with a wider subnet to fix this.
Emergency Relief: Scale down non-critical deployments. Deleting 20 replicas of a dev app can often buy you enough breathing room to fix the underlying network.

Step 1: Locate the Bottleneck

Is the problem isolated to one node or the entire network?

Check individual Node capacity

A single node might fail if it hits a hardware limit. For example, an AWS t3.medium instance supports only 3 ENIs and 6 IPs per ENI. If you try to run 18 pods on it, you'll hit the ceiling. Check your CNI logs to see if it's struggling to allocate local resources:

kubectl logs -n kube-system -l k8s-app=aws-node
# For Calico users
kubectl logs -n kube-system -l k8s-app=calico-node

Check Subnet-wide exhaustion

If every node in the cluster is throwing errors, your VPC subnet is likely tapped out. A /24 subnet only provides 251 usable addresses after AWS or Azure takes their share. Check your cloud console's VPC section to see the "Available IP Address" count.

Step 2: Apply the Fix

Scenario A: AWS VPC CNI (EKS)

EKS maps pod IPs directly from your VPC subnets. This makes networking fast but consumes IPs quickly.

Fix 1: Tweak WARM_IP_TARGETThe CNI keeps a "warm" pool of IPs ready for fast pod startup. If this number is too high, a single node might reserve 10 IPs even if it only runs 2 pods. You can lower this by editing the aws-node DaemonSet:

kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=3

Fix 2: Add a Secondary CIDRIf your VPC is truly out of space, attach a secondary CIDR block (like 100.64.0.0/16) to the VPC. Then, configure the CNI to use this new range for pods via Custom Networking.

Scenario B: Calico IPPools

Calico divides its CIDR into blocks (usually /26, which allows 64 IPs) and hands them to nodes. If you have many small nodes, you might run out of blocks even if the total IP count looks fine.

Check your pool usage:

calicoctl get ippool -o wide

If the usage is near 100%, define and apply a new IPPool with a larger range:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: expansion-pool
spec:
  cidr: 192.168.64.0/18
  ipipMode: Always
  natOutgoing: true

Step 3: Verify the Recovery

After expanding the range, watch the pod status. The FailedCreatePodSandBox errors should vanish from your event log.

# Watch for pods moving from Pending to Running
kubectl get pods -A -w | grep -v Running

Confirm the CNI is actually working by tailing the logs for success messages:

kubectl logs -n kube-system [CNI_POD_NAME] | grep "Successfully assigned IP"

Pro-Tips for Prevention

IP exhaustion is usually just a math error. Before launching a new node group, calculate your peak pod density. Remember to account for rolling updates. During a deployment, you might temporarily double your pod count, requiring twice the IPs.

I use a subnet calculator to visualize how many pods a /24 or /22 can actually hold. Don't forget to subtract node IPs and reserved addresses from that total. I personally use the Subnet Calculator on ToolCraft. It's fast, works in the browser, and keeps your network data private. It's a great way to double-check your CIDR math before applying a new IPPool.

Clearing "Zombie" IPs

Sometimes, a node crashes and the CNI fails to release its IPs. This creates "zombie" allocations that take up space. Restarting the CNI pods usually triggers a resync that cleans these up:

kubectl rollout restart ds aws-node -n kube-system

On bare metal, check /var/lib/cni/networks/ on the host node. You might find stale files representing old allocations that didn't get deleted.

Final Thoughts

Expanding your IP range solves the immediate crash, but don't forget your firewall. Ensure your Security Groups or on-prem firewalls allow traffic from the new CIDR range. If you forget this step, your pods will start, but they won't be able to talk to anything.