kube-state-metrics
kube-state-metrics copied to clipboard
kubernetes
kube_cronjob_next_schedule_time in the past when cronjob timezone specified
What happened:
Defined a timezone for a cronjob:
timeZone: "Europe/London"
Now kube_cronjob_next_schedule_time is always returning the last schedule time rather than the next one.
What you expected to happen: kube_cronjob_next_schedule_time should be the next scheduled time.
How to reproduce it (as minimally and precisely as possible): Define a cronjob with non-UTC timezone (may only happen on zones east of UTC). Wait for the job to execute and then check kube_cronjob_next_schedule_time - you will see it's in the past.
- kube-state-metrics version: 2.10
- Kubernetes version (use
kubectl version): 1.27 - Cloud provider or hardware configuration: Azure AKS
The source code is here: https://github.com/kubernetes/kube-state-metrics/blob/b91dd311397047f15aae4f6dacba0708f24cf446/internal/store/cronjob.go#L238-L263
Could you dive into it and help debug if you are available?
I'm not a go developer, but I suspect that what's happening is that j.Spec.Schedule doesn't include the timezone, so the cron library is interpreting it in local time (UTC). So if this code were to run just after the cronjob had run (in UTC+1), it would say the next scheduled time was still 1 hour away.
The cronjob library expects the timezone to be specified like this:
schedule, err := cron.ParseStandard("TZ=America/Los_Angeles 0 2 * * 0")
or
loc, _ := time.LoadLocation("Europe/Paris") cronProcess := cron.New(cron.WithLocation(loc))
I'm not sure if this is being done.
Hi @paulclark222, j.Spec does have a field named timezone, which hasn't been used
https://github.com/kubernetes/api/blob/master/batch/v1/types.go#L637
Could you ramp up golang and try to fix this issue, if you are available and have enough time?
The code for the metric predates the support for timezone, so it is not taken into account. We would need to had support for it in ksm.
/help
@dgrisonnet: This request has been marked as needing help from a contributor.
Guidelines
Please ensure that the issue body includes answers to the following questions:
- Why are we solving this issue?
- To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
- Does this issue have zero to low barrier of entry?
- How can the assignee reach out to you for help?
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.
In response to this:
The code for the metric predates the support for timezone, so it is not taken into account. We would need to had support for it in ksm.
/help
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
I encountered the same issue after configuring the timezone. Any recent updates or ETA on a resolution? Thanks!
+1 Really an issue when you are trying to create alerts if cronjobs are running longer than expected.
I have a fix in https://github.com/kubernetes/kube-state-metrics/pull/2376
Can you test if this resolves your issue?