When Your Deployment Hits a WallYour CI/CD pipeline is running smoothly until a kubectl apply suddenly hangs for 30 seconds and fails. Instead of a successful deployment, the Kubernetes API server throws a specific internal error that blocks all new Ingress resources. It looks like this:
Error from server (InternalError): Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": service "ingress-nginx-controller-admission" not found
This error typically crops up after you've tried to uninstall or upgrade the NGINX Ingress Controller. It’s frustrating because it references a service that no longer exists, yet it has the power to stop your infrastructure updates in their tracks.
The Ghost in the API ServerAdmission controllers act as gatekeepers for your cluster. Before any object is saved to the etcd database, these controllers vet the request. NGINX uses a ValidatingWebhookConfiguration to ensure your Ingress YAML doesn't have syntax errors that could break the controller.
The breakdown happens when you delete the NGINX pods or services but leave the global webhook configuration behind. Because the default failurePolicy is set to Fail, the API server refuses to accept any Ingress changes if it cannot get a 'green light' from the validator. It chooses to 'fail closed' to prevent potentially invalid configurations from entering the cluster.
Solution 1: Scrubbing the Orphaned WebhookIf you have uninstalled NGINX Ingress or are moving to a different controller, you need to remove the stale configuration. This is the most common fix for 90% of users.
1. Find the culpritRun this command to see which webhooks are currently active in your cluster:
kubectl get validatingwebhookconfigurations
Look for an entry named ingress-nginx-admission. In some older Helm charts, it might just be named ingress-nginx.
2. Delete the configurationRemove the global object. Since this isn't namespaced, you can run it from anywhere:
kubectl delete validatingwebhookconfigurations ingress-nginx-admission
Once deleted, the API server will stop trying to call the non-existent admission service. Your kubectl apply commands should start working immediately.
Solution 2: Repairing a Broken InstallationSometimes you didn't mean to delete NGINX, but a Helm upgrade went sideways. If the Ingress pods are running but the admission service is missing, you need to restore the connection.
1. Verify the serviceCheck if the admission service exists in your ingress-nginx namespace:
kubectl get svc -n ingress-nginx
2. Force a Helm refreshIf the service is missing, trigger Helm to recreate the missing components. Use this command to ensure the webhook flags are explicitly set:
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \n --namespace ingress-nginx \n --set controller.admissionWebhooks.enabled=true
This command synchronizes the webhook configuration with the actual state of your services, fixing the 'service not found' disconnect.
Solution 3: The Emergency BypassSuppose it's 2 AM and you need to push a critical fix, but you can't figure out why the admission service is down. You can temporarily tell Kubernetes to ignore the webhook failure.
Open the configuration for editing:
kubectl edit validatingwebhookconfigurations ingress-nginx-admission
Locate the failurePolicy and change it from Fail to Ignore:
webhooks:\n- name: validate.nginx.ingress.kubernetes.io\n failurePolicy: Ignore\n # ...
After saving, the API server will still try to call the webhook. However, when the call fails, it will simply skip the validation and allow your resource to be created.
Verification and PreventionConfirm the fix by applying a test Ingress or your original deployment. If the command returns configured or unchanged without a delay, the blockage is gone. To avoid this in the future, follow these two rules:
- Never manually delete namespaces: Always use
helm uninstall. Manual deletion often skips global resources like webhooks and CRDs.- Monitor Webhook Latency: Use Prometheus to trackadmissions_webhook_admission_duration_seconds. Sudden spikes often signal that your admission controller is struggling before it actually fails.