What will happen to evicted pods in kubernetes?
KubernetesKubernetes Problem Overview
I just saw some of my pods got evicted by kubernetes. What will happen to them? just hanging around like that or I have to delete them manually?
Kubernetes Solutions
Solution 1 - Kubernetes
A quick workaround I use, is to delete all evicted pods manually after an incident. You can use this command:
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete pods \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c
Solution 2 - Kubernetes
To delete pods in Failed state in namespace default
kubectl -n default delete pods --field-selector=status.phase=Failed
Solution 3 - Kubernetes
Evicted pods should be manually deleted. You can use following command to delete all pods in Error
state.
kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -
Solution 4 - Kubernetes
Depending on if a soft or hard eviction threshold that has been met, the Containers in the Pod will be terminated with or without grace period, the PodPhase
will be marked as Failed
and the Pod deleted. If your Application runs as part of e.g. a Deployment, there will be another Pod created and scheduled by Kubernetes - probably on another Node not exceeding its eviction thresholds.
Be aware that eviction does not necessarily have to be caused by thresholds but can also be invoked via kubectl drain
to empty a node or manually via the Kubernetes API.
Solution 5 - Kubernetes
The bellow command delete all failed pods from all namespaces
kubectl get pods -A | grep Evicted | awk '{print $2 " -n " $1}' | xargs -n 3 kubectl delete pod
Solution 6 - Kubernetes
To answer the original question: the evicted pods will hang around until the number of them reaches the terminated-pod-gc-threshold
limit (it's an option of kube-controller-manager and is equal to 12500 by default), it's by design behavior of Kubernetes (also the same approach is used and documented for Jobs - https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup). Keeping the evicted pods pods around allows you to view the logs of those pods to check for errors, warnings, or other diagnostic output.
Solution 7 - Kubernetes
One more bash command to delete evicted pods
kubectl get pods | grep Evicted | awk '{print $1}' | xargs kubectl delete pod
Solution 8 - Kubernetes
Just in the case someone wants to automatically delete all evicted pods for all namespaces:
- Powershell
Foreach( $x in (kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name)) {kubectl delete po $x --all-namespaces }
- Bash
kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name | xargs kubectl delete po --all-namespaces
Solution 9 - Kubernetes
Kube-controller-manager
exists by default with a working K8s installation. It appears that the default is a max of 12500 terminated pods before GC kicks in.
Directly from the K8s documentation: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#kube-controller-manager
> --terminated-pod-gc-threshold int32 Default: 12500
> Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled.
Solution 10 - Kubernetes
In case you have pods with a Completed
status that you want to keep around:
kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -
Solution 11 - Kubernetes
Another way still with awk
.
To prevent any human error that could make me crazy (deleting desirable pods), I check before the result of the get pods
command :
kubectl -n my-ns get pods --no-headers --field-selector=status.phase=Failed
If that looks good, here we go :
kubectl -n my-ns get pods --no-headers --field-selector=status.phase=Failed | \
awk '{system("kubectl -n my-ns delete pods " $1)}'
Same thing with pods of all namespaces.
Check :
kubectl get -A pods --no-headers --field-selector=status.phase=Failed
Delete :
kubectl get -A pods --no-headers --field-selector status.phase=Failed | \
awk '{system("kubectl -n " $1 " delete pod " $2 )}'
Solution 12 - Kubernetes
OpenShift equivalent of Kalvin's command to delete all 'Evicted' pods:
eval "$(oc get pods --all-namespaces -o json | jq -r '.items[] | select(.status.phase == "Failed" and .status.reason == "Evicted") | "oc delete pod --namespace " + .metadata.namespace + " " + .metadata.name')"
Solution 13 - Kubernetes
To delete all the Evicted
pods by force, you can try this one-line command:
$ kubectl get pod -A | sed -nr '/Evicted/s/(^\S+)\s+(\S+).*/kubectl -n \1 delete pod \2 --force --grace-period=0/e'
Tips: use the p
modifier of s
command of sed
instead of e
will just print the real command to do the deletion job:
$ kubectl get pod -A | sed -nr '/Evicted/s/(^\S+)\s+(\S+).*/kubectl -n \1 delete pod \2 --force --grace-period=0/p'
Solution 14 - Kubernetes
below command will get all evicted pods from the default namespace and delete them
kubectl get pods | grep Evicted | awk '{print$1}' | xargs -I {} kubectl delete pods/{}
Solution 15 - Kubernetes
Here is the 'official' guide for how to hard code the threshold(if you do not want to see too many evicted pods): kube-controll-manager
But a known problem is how to have kube-controll-manager installed...