The Monday Morning OutageYou sit down, grab your coffee, and run a routine kubectl get nodes. Instead of a healthy cluster list, your terminal spits back a TLS error:
Unable to connect to the server: x509: certificate has expired or is not yet valid
This usually happens exactly 365 days after you first initialized the cluster. Because the internal certificates have hit their expiration date, the API server now rejects every request. Your applications might still be running, but your ability to manage them is completely frozen.
Why Kubernetes Certificates ExpireWhen you set up a cluster using kubeadm, it generates a set of certificates in /etc/kubernetes/pki. By default, these are valid for only one year. While modern Kubernetes versions attempt to rotate some Kubelet certificates automatically, the core control plane certificates often require a manual refresh if you haven't performed a cluster upgrade recently.
The API server relies on these certificates for secure communication. Once that one-year clock runs out, the TLS handshake fails, and kubectl loses its permission to talk to the cluster.
Step 1: Audit Your Certificate ExpiryFirst, let's see exactly which certificates died. Log into your master (control-plane) node and check the status:
sudo kubeadm certs check-expiration
Note: If you're on an older version (pre-v1.20), use sudo kubeadm alpha certs check-expiration instead.
Look at the RESIDUAL TIME column. If you see 0d or negative values for entries like admin.conf or apiserver, you've found the problem. Even if they have 2 or 3 days left, it's time to renew them now before everything breaks.
Step 2: Renew Everything at OnceThe most efficient fix is to renew every certificate managed by the cluster. Execute this on your master node:
sudo kubeadm certs renew all
This command updates the .crt and .key files in /etc/kubernetes/pki. It also updates the embedded certificates within your .conf files. However, your cluster isn't fixed yet—your local user profile still has the old credentials.
Step 3: Update Your KubeconfigYour ~/.kube/config file is a snapshot of the old, expired certificate. If you don't update it, kubectl will keep trying to use the dead credentials. Overwrite your local config with the fresh one generated by the renewal:
# For the root or current user on the master node
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
If you manage the cluster from a remote laptop, copy the contents of the new /etc/kubernetes/admin.conf to your local machine's config file. Without this step, you'll continue seeing the x509 error even though the server is healthy.
Step 4: Force the Control Plane to RestartThe API server and Controller Manager are still running in memory using the old, expired certificates. You need to kick them so they load the new files. Since these are static pods, the most reliable method is to move their manifest files out of the deployment folder and back in.
# Move manifests to a temp folder to stop the pods
sudo mv /etc/kubernetes/manifests/*.yaml /tmp/
# Wait about 30 seconds for the Kubelet to terminate the processes
sleep 30
# Move them back to trigger a restart
sudo mv /tmp/*.yaml /etc/kubernetes/manifests/
Wait a minute for the API server to stabilize. You can watch the process with watch docker ps or watch crictl ps to ensure the containers are back up.
Verifying the FixOnce the services are back online, test your connection:
kubectl get nodes
If you see your nodes listed as Ready, the crisis is over. Run sudo kubeadm certs check-expiration one last time. You should now see 364 days of remaining life across the board.