stern icon indicating copy to clipboard operation
stern copied to clipboard

Make `--condition=Ready=false` match running jobs

Open felipecrs opened this issue 10 months ago • 6 comments

This PR adds a special handling for --condition=Ready=false for jobs.

Jobs otherwise reports as Ready=true as soon as they are started, meaning no jobs would ever be part of the --condition=Ready=false list, which makes it very difficult to track progress of normal pods alongside jobs using stern.

This PR changes the code to consider jobs not to be ready until they are completed. Thus, if the job is running, it is not ready. Which I think makes total sense.

What do you think?

felipecrs avatar Feb 19 '25 23:02 felipecrs

I will check the PR this weekend.

superbrothers avatar Mar 18 '25 08:03 superbrothers

Jobs otherwise reports as Ready=true as soon as they are started, meaning no jobs would ever be part of the --condition=Ready=false list, which makes it very difficult to track progress of normal pods alongside jobs using stern.

I don't understand your problem correctly. Please tell me the steps I can take to reproduce your problem.

Does this mean that the log is not output with the following command?

$ kubectl create job my-job --image=busybox -- sh -c "date"
$ stern . --condition=ready=false --tail=0
+ my-job-jh2cj › my-job
- my-job-jh2cj › my-job

The --condition flag is also used with the --no-follow flag. In this case, the log is output as follows.

$ stern . --condition=ready=false --no-follow
+ my-job-gmqws › my-job
my-job-gmqws my-job Sun Mar 23 12:41:43 UTC 2025
- my-job-gmqws › my-job

This change in PR will break existing behavior, so it is difficult to accept it as it is.

superbrothers avatar Mar 23 '25 13:03 superbrothers

@superbrothers, thanks a lot for looking into it!

Does this mean that the log is not output with the following command?

Yeah, that's right. Logs for jobs are never shown when using --condition=ready=false.

The --condition flag is also used with the --no-follow flag. In this case, the log is output as follows.

I'm not sure what's happening here. Is this with current stern or with this PR? If it's with current stern, this is probably a bug.

This change in PR will break existing behavior, so it is difficult to accept it as it is.

Well, the --condition flag is new, and it would be a welcome change by all users of it, I suppose.

But if not it's not tolerable, I will understand. In that case, would you have any suggestion on how to present this feature? I could only think of something like --uncondition-jobs.

felipecrs avatar Mar 23 '25 17:03 felipecrs

--condition flag must be used with either --tail=0 or --no-follow.

When --condition is used with --tail=0, in the above steps, when the condition is met, there is no output because the job pod has already terminated and there is no log output. This is not a bug.

Please be more specific about the use cases you would like to achieve so that we can better consider changes. I would prefer to be able to perform the steps you present on my local cluster.

superbrothers avatar Mar 24 '25 00:03 superbrothers

@superbrothers, consider the following job:

apiVersion: batch/v1
kind: Job
metadata:
  name: test-job
spec:
  completions: 1
  backoffLimit: 999
  template:
    metadata:
      labels:
        app: test-job
    spec:
      terminationGracePeriodSeconds: 1
      restartPolicy: OnFailure
      containers:
        - name: test
          image: bash
          command:
            - bash
            - -c
            - |
              count=0
              while true; do
                echo "World Hello: $count"
                count=$((count+1))
                if [[ $count -eq 10 ]]; then
                  exit 1
                fi
                sleep 1
              done

And consider the following command:

stern . --condition=ready=false --tail=0

Stern would never capture logs for this pod:

https://github.com/user-attachments/assets/d433f396-77fa-45cc-874c-942f8773b23a

felipecrs avatar Mar 24 '25 01:03 felipecrs

I want to use Stern in a CI pipeline to display logs during a deployment for the pods that aren't yet ready. Because of this issue, Stern will never display logs for jobs.

felipecrs avatar Mar 24 '25 01:03 felipecrs