kubectl
kubectl copied to clipboard
kubernetes
how to kubectl wait for job completion regardless of success or failure
We are running AKS and use Azure DevOps with kubectl task to deploy our application. Before we deploy the actual application, we are running a data migration job. We want to wait for this job in the Azure DevOps pipeline, however the only option so far is to:
kubectl wait --for=condition=complete --timeout=600s job/somejob
In this case the pipeline task would only exit if the job has completed or the timeout has elapsed. In the case the job fails we are left with just waiting for timeout. I looked at using kubectl wait --for=jsonpath but there is no single value (and i mean single as in singular, not a combination of different values) in the output returned by kubectl that would indicate the job has finished, regardless of whether it succeeded or failed. And due to kubectl not supporting and/or conditions in jsonpath I am not able to evaluate the job completion based on multiple values. The only field that represents true completion is when .status.completionTime is set. I tried just out of curiosity doing kubectl wait --for=jsonpath={.status.completionTime}, and the task did indeed wait for the job once .status.completionTime was set, however docs suggest the condition would be met if the value I look for is True, but obviously completionTime is not. Is there any docs that would indicate how wait --for=jsonpath behaves on the fields that are set dynamically (like .status.completionTime) during the job execution?
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:
/sig <group-name>/wg <group-name>/committee <group-name>
Please see the group list for a listing of the SIGs, working groups, and committees available.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
This issue is currently awaiting triage.
If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Just noticed from the docs that "The completion time is only set when the job finishes successfully.". Hence this will not work in my case when the job fails. Is there other singular value in the job status that represents the completion (finish) of the job, regardless of whether it failed or succeeded?
Related issue https://github.com/kubernetes/kubectl/issues/1224
See possible workaround in this comment https://github.com/kubernetes/kubectl/issues/1224#issuecomment-1562346598
Relevant Stack Overflow https://stackoverflow.com/questions/55073453/wait-for-kubernetes-job-to-complete-on-either-failure-success-using-command-line
/triage accepted There are plans to implement this feature, but until that feature is implemented we recommend that you either implement the scripts in Brian's links, or you can use Brian's plugin as a workaround.
As @mpuckett159 mentioned, I created a plugin to help with this.
Install:
kubectl krew update && kubectl krew install wait-job
Usage:
kubectl wait-job somejob
Output with be either:
Job completed successfullywith exit code 0Job failedwith exit code 1
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
