What will happen to evicted pods in the quernets?

Question

I just saw some of my pods being evicted by the quernets. What will happen to them? just hung around, or should I remove them manually?

Answer 1

The quick workaround I'm using is to manually remove all the evicted modules after the incident. You can use this command:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete pods \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c 

Answer 2

To remove modules in the Failed state in the default namespace

 kubectl -n default delete pods --field-selector=status.phase=Failed 

Answer 3

Depending on whether a soft or hard eviction threshold is met, containers in the Pod will be terminated with or without a grace period, PodPhase will be marked as Failed and Pod removed. If your application starts as part of, for example, a deployment, there will be another Pod created and scheduled by Kubernetes - probably on a different node that does not exceed the threshold values ​​for the exception.

Keep in mind that an exception does not have to be triggered by thresholds, but it can also be kubectl drain via kubectl drain for a kubectl drain node or manually through the Kubernetes API .

Answer 4

Evicted pods must be removed manually. You can use the following command to remove all modules in the Error state.

 kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f - 

Answer 5

OpenShift is the equivalent of the Kalvin command to remove all "output" modules:

 eval "$(oc get pods --all-namespaces -o json | jq -r '.items[] | select(.status.phase == "Failed" and .status.reason == "Evicted") | "oc delete pod --namespace " + .metadata.namespace + " " + .metadata.name')" 

Answer 6

If you have modules with the status " Completed that you want to save:

 kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f - 

Answer 7

Just in case, someone wants to automatically remove all the evicted modules for all namespaces:

• Powershell

  Foreach( $x in (kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name)) {kubectl delete po $x --all-namespaces } 

• Bash

 kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name | xargs kubectl delete po --all-namespaces 

Answer 8

Kube-controller-manager exists by default with a working installation of K8s. It looks like the default is a maximum of 12,500 completed pods before the GC takes effect.

Directly from the K8s documentation: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#kube-controller-manager

--terminated-pod-gc-threshold int32 Default: 12500
The number of completed modules that may exist before the completed module garbage collector begins to delete completed modules. If <= 0, the stopped pod garbage collector is disabled.

Answer 9

Here is the "official" guide to hard coding the threshold (if you don't want to see too many evicted modules): kube-controll-manager

But a known issue is how to install kube-controll-manager ...

Answer 10

Another bash command to remove evicted modules.

 kubectl get pods | grep Evicted | awk '{print $1}' | xargs kubectl delete pod