Decoding the 503 Error
You hit your application's URL, but instead of a dashboard, you get a cold 503 Service Temporarily Unavailable screen. If you're using NGINX Ingress, your controller logs will likely show a specific culprit:
[error] 197#197: *3586884 Service "default/my-service" does not have any active endpoints
This message is actually helpful. It tells you that while the Ingress found your Service, the Service couldn't find any healthy Pods to handle the request.
The Root Cause
Kubernetes routing works like a chain: Ingress points to a Service, and that Service points to a set of Pods. For this to function, the Service maintains an Endpoints list—a real-time directory of IP addresses for healthy Pods.
A 503 error triggers when that directory is empty. Usually, the breakdown happens because of a label mismatch, a failing health check, or a simple port typo.
Step-by-Step Troubleshooting
1. Audit the Endpoints Resource
Start by verifying if the bridge between the Service and Pods is broken. Run this command:
kubectl get endpoints my-service -n <your-namespace>
Look at the ENDPOINTS column. If you see <none>, the traffic has nowhere to go. This confirms the issue lies with your Pods or how the Service selects them, rather than the Ingress itself.
2. Hunt for Label Mismatches
Labels are the glue of Kubernetes. If your Service looks for app: payment-api but your Deployment labels Pods as app: payment-gateway, the Service will never find them. It’s a common copy-paste error.
Check what the Service is looking for:
kubectl describe svc my-service -n <your-namespace> | grep Selector
Then, list the actual labels on your Pods:
kubectl get pods -n <your-namespace> --show-labels
The Pod must have every label listed in the Service selector. If the Service specifies three labels and the Pod only has two, it won't receive traffic.
3. Diagnose Failing Readiness Probes
A Pod can be "Running" but not "Ready." If its readiness probe fails, Kubernetes removes its IP from the Endpoints list to prevent sending traffic to a broken container.
Check the READY column for your Pods:
kubectl get pods -n <your-namespace>
If you see 0/1, the Pod is alive but failing health checks. For details, run kubectl describe pod <pod-name> and check the "Events" section at the bottom. You might see: Readiness probe failed: HTTP probe failed with statuscode: 500. This often happens if an app (like a Java Spring Boot service) takes 45 seconds to start, but the probe starts checking after only 5 seconds.
4. Verify Port Mapping
Typos in port numbers are a silent killer. If your application container listens on port 8080, but your Service sends traffic to targetPort: 80, the connection will time out.
Ensure your Service targetPort matches the containerPort in your Deployment:
# Service
ports:
- port: 80
targetPort: 8080 # Must match below
# Deployment
containers:
- name: my-app
ports:
- containerPort: 8080 # Must match above
Confirming the Fix
After updating your labels or probes, check the Endpoints list again. You should see a list of internal IP addresses like 10.244.1.45:8080. To be 100% sure, run a quick connectivity test from within the cluster:
kubectl run curl-test --image=curlimages/curl -i --tty --rm -- curl http://my-service:80
A 200 OK response means your 503 error is officially resolved.
Best Practices
- Standardize Labels: Use the
app.kubernetes.io/namestandard to keep your selectors consistent across teams. - Graceful Probe Delays: Set
initialDelaySecondsto match your app's actual startup time. Giving a slow app 30 seconds of breathing room prevents unnecessary 503s. - Automated Alerts: Use tools like Prometheus to alert you when
kube_endpoint_address_availabledrops to zero.