The Error Message
Preparing a Kubernetes node for maintenance—whether for a kernel patch, hardware replacement, or a cluster scale-down—usually starts with kubectl drain. But more often than not, the process halts with this specific error:
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-xxxx, kube-system/calico-node-xxxx
This happens because kubectl has spotted pods managed by a DaemonSet, such as kube-proxy, calico-node, or a logging agent like Fluentd. It refuses to proceed because it doesn't want to terminate system-level services without your explicit permission.
Root Cause: Why Kubernetes Protects DaemonSets
The kubectl drain command works by evicting pods so they can be rescheduled on other nodes. DaemonSets are the exception to this rule. They are designed to run exactly one copy of a pod on every node in the cluster.
If Kubernetes "evicted" a DaemonSet pod, there would be nowhere for it to go. Every other node already has its own copy. Since these pods often handle critical networking or monitoring, Kubernetes forces you to acknowledge that these services will stop on this specific node and won't migrate elsewhere during the downtime.
Step-by-Step Fix
1. The Standard Workaround: --ignore-daemonsets
The fastest fix is usually the one suggested in the error itself. By adding this flag, you grant Kubernetes permission to terminate DaemonSet pods on the target node without trying to reschedule them elsewhere.
# Replace <node-name> with your actual node ID
kubectl drain <node-name> --ignore-daemonsets
This is safe for 99% of clusters. Once the node reboots and you uncordon it, the DaemonSet controller will automatically spin those pods back up.
2. Dealing with Local Storage (emptyDir)
After bypassing the DaemonSet error, you might run into a second hurdle: cannot delete Pods with local storage. This occurs when pods use emptyDir volumes for scratch space or caching. Because emptyDir data is wiped when a pod is deleted, Kubernetes pauses to prevent accidental data loss.
If you're sure the local data is ephemeral, use both flags together:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
3. Handling "Naked" Pods with --force
Occasionally, you might have pods that weren't created by a controller like a Deployment or StatefulSet. These "naked" pods have no way to recreate themselves if they are deleted. If you are certain these pods are disposable or you have manual backups, use the --force flag to clear them out.
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force
Verification: Confirming the Node is Ready
Check the status of your node immediately after the command finishes. You are looking for the SchedulingDisabled status.
kubectl get nodes
Expected output should look like this:
NAME STATUS ROLES AGE VERSION
worker-node-1 Ready,SchedulingDisabled worker 12d v1.28.2
Run a quick check to see what's left on the node. Only DaemonSet pods should remain, and they will typically be in a Terminating or Running state depending on how they were handled.
kubectl get pods -A -o wide | grep worker-node-1
Prevention and Best Practices
Configure Pod Disruption Budgets (PDB)
Don't let a drain command accidentally take your entire application offline. Define a PDB to tell Kubernetes the minimum number of replicas that must remain active during maintenance. If a drain would violate this limit, the command will fail safely.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: my-web-app
Set Drain Timeouts
In automated pipelines, a single stuck pod can hang your entire CI/CD process for hours. Use a timeout to ensure the process fails fast if a pod refuses to terminate due to complex finalizers or hardware issues.
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=300s
Return the Node to Service
Once your maintenance tasks are finished, tell Kubernetes the node is open for business again. This step is easy to forget but essential for balancing your cluster load.
kubectl uncordon <node-name>
After uncordoning, your DaemonSet pods will be the first to stabilize, followed by standard workloads moving back to the node.