How to list Kubernetes recently deleted pods?

Kubernetes

Kubernetes Problem Overview


Is there a way to get some details about Kubernetes pod that was deleted (stopped, replaced by new version).

I am investigating bug. I have logs with my pod name. That pod does not exist anymore, it was replaced by another one (with different configuration). New pod resides in same namespace, replication controller and service as old one.

Commands like

kubectl  get pods
kubectl  get pod <pod-name> 

work only with current pods (live or stopped).

How I could get more details about old pods? I would like to see

  1. when they were created
  2. which environment variables they had when created
  3. why and when they were stopped

Kubernetes Solutions


Solution 1 - Kubernetes

As of today, kubectl get pods -a is deprecated, and as a result you cannot get deleted pods.

What you can do though, is to get a list of recently deleted pod names - up to 1 hour in the past unless you changed the ttl for kubernetes events - by running:

kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

You can then investigate further issues within your logging pipeline if you have one in place.

Solution 2 - Kubernetes

as far as i know you may not get the pod details once the pod is deleted. can i know what is the usecase?

example:

  1. if a pod created using - kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/false you will have a pod with status termiated:error
  2. if a pod is created using - kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/true you will have pod with status terminated:Complted
  3. if container in a pods restarts: the pod will be alive and you can get the logs of previous container (only the previous container) using kubectl logs --container < container_name > --previous=true < pod_name >
  4. if you doing an upgrade of you app and you are creating pods using deployments. if the update deployment "say a new image". pod will be terminated and new pod will be created. you can get the pod details from depoyment yaml. if you want to get details of previous pod you have see "spec" section of previous deployment yaml

Solution 3 - Kubernetes

You can try kubectl logs --previous to list the logs of a previously stopped pod

http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/

You may also want to check out these debugging tips http://kubernetes.io/docs/user-guide/debugging-pods-and-replication-controllers/

Solution 4 - Kubernetes

There is a way to find out why pods were deleted and who deleted them. The only way to find out something is to set the ttl for k8s to be greater than the default 1h and search through the events:

kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

Solution 5 - Kubernetes

If your container has previously crashed, you can access the previous container’s crash log with:

kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}

Solution 6 - Kubernetes

There is this flag:

> -a, --show-all=false: When printing, show all resources (default hide terminated pods.)

But this may not help in all cases of old pods.

Solution 7 - Kubernetes

kubectl get pods -a    

you will get the list of running pods and the terminated pods in case you are searching for this

Solution 8 - Kubernetes

If you want to see all the previously deleted pods and you are trying to fetch the previous pods.

Command line: kubectl get pods

in which you will get all the pod details, because every service has one or more pods and they have unique ip address

Here you can check the lifecycle of pods and what phases of pod has. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle<a>

and you can see the previous pod logs by typing a command: kubectl logs --previous

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionBartosz BilickiView Question on Stackoverflow
Solution 1 - KubernetesiomvView Answer on Stackoverflow
Solution 2 - KubernetesSandeep kumar singhView Answer on Stackoverflow
Solution 3 - KubernetesMatt RickardView Answer on Stackoverflow
Solution 4 - KubernetesPaul ChibulcuteanuView Answer on Stackoverflow
Solution 5 - KubernetesSunil GajulaView Answer on Stackoverflow
Solution 6 - KubernetesmanojldsView Answer on Stackoverflow
Solution 7 - KubernetesHossam KhalilView Answer on Stackoverflow
Solution 8 - KubernetesHushenView Answer on Stackoverflow