Tell when Job is Complete
KubernetesKubectlKubernetes JobsKubernetes Problem Overview
I'm looking for a way to tell (from within a script) when a Kubernetes Job has completed. I want to then get the logs out of the containers and perform cleanup.
What would be a good way to do this? Would the best way be to run kubectl describe job <job_name>
and grep for 1 Succeeded
or something of the sort?
Kubernetes Solutions
Solution 1 - Kubernetes
Since version 1.11, you can do:
kubectl wait --for=condition=complete job/myjob
and you can also set a timeout:
kubectl wait --for=condition=complete --timeout=30s job/myjob
Solution 2 - Kubernetes
You can visually watch a job's status with this command:
kubectl get jobs myjob -w
The -w
option watches for changes. You are looking for the SUCCESSFUL
column to show 1
.
For waiting in a shell script, I'd use this command:
until kubectl get jobs myjob -o jsonpath='{.status.conditions[?
(@.type=="Complete")].status}' | grep True ; do sleep 1 ; done
Solution 3 - Kubernetes
You can use official Python kubernetes-client
.
https://github.com/kubernetes-client/python
Create new Python virtualenv:
virtualenv -p python3 kubernetes_venv
activate it with
source kubernetes_venv/bin/activate
and install kubernetes client with:
pip install kubernetes
Create new Python script and run:
from kubernetes import client, config
config.load_kube_config()
v1 = client.BatchV1Api()
ret = v1.list_namespaced_job(namespace='<YOUR-JOB-NAMESPACE>', watch=False)
for i in ret.items:
print(i.status.succeeded)
Remember to set up your specific kubeconfig
in ~/.kube/config
and valid value for your job namespace -> '<YOUR-JOB-NAMESPACE>'
Solution 4 - Kubernetes
I would use -w
or --watch
:
$ kubectl get jobs.batch --watch
NAME COMPLETIONS DURATION AGE
python 0/1 3m4s 3m4s
Solution 5 - Kubernetes
It either one of these queries with kubectl
kubectl get job test-job -o jsonpath='{.status.succeeded}'
or
kubectl get job test-job -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}'
Solution 6 - Kubernetes
Adding the best answer, from a comment by @Coo, If you add a -f
or --follow
option when getting logs, it'll keep tailing the log and terminate when the job completes or fails. The $#
status code is even non-zero when the job fails.
kubectl logs -l job-name=myjob --follow
One downside of this approach, that I'm aware of, is that there's no timeout option.
Another downside is the logs call may fail while the pod is in Pending
(while the containers are being started). You can fix this by waiting for the pod:
# Wait for pod to be available; logs will fail if the pod is "Pending"
while [[ "$(kubectl get pod -l job-name=myjob -o json | jq -rc '.items | .[].status.phase')" == 'Pending' ]]; do
# Avoid flooding k8s with polls (seconds)
sleep 0.25
done
# Tail logs
kubectl logs -l job-name=myjob --tail=400 -f