cronjobber Cronjobber does not work with slim tzdata format

Hello,

I have a two CronJob the one set to 0 1 * * * in Europe/Paris which is running at 3 CEST, the other one scheduled at 0 3 * * * in Europe/Paris ran at 5 CEST...

Do you have any idea why it's not running at the time it was scheduled ?

Thanks,

Oct 23 '20 07:10 scorsi

Hello,

Same issue here. We normally scale up our pods in the evening, yet they have been triggered in the morning today. I found the following text in the logs:

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Best, u6f6o

Oct 23 '20 12:10 u6f6o

Hello, @u6f6o and I have the exact same problem.

We have 4 different namespaces with 2 jobs each. Of those jobs, 4 are scheduled for 19:50 PM and 4 are scheduled for 20:30 PM, each day.

Today, they all ran at exactly the same time (10:44 AM). Maybe it is a coincidence, but the updatetz container logged the right before all jobs ran:

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Our local time zone is CEST (+0200), thus the database update was logged at 10:43:58 AM local time.

Oct 23 '20 12:10 pscheid92

May be related to #24. Actually don't have the workload to investigate deeper. Switched to default Kubernetes CronJob while the issue is not fixed/investigated.

Oct 23 '20 12:10 scorsi

Running into the same problem here. Now my TZCronJobs won't start at all, at least not during my specified time. I haven't been able to debug the tzdb inside the container, has there been an issue with the update?

Oct 23 '20 13:10 obitech

Sorry about this unexpected behaviour, and I hope it did not result in too much chaos. Can you all please confirm you are using the latest cronjobber-updatetz release (v0.1.1)? This helps me narrow my search.

Oct 23 '20 19:10 hiddeco

Yes, we (@u6f6o, @obitech and me) used the v0.1.1 release (with sidecar.enabled=true; installed per Helm chart).

Oct 23 '20 20:10 pscheid92

Just noticed this is happening for us as well. Running cronjobber:0.2.0 and cronjobber-updatetz:0.1.0. Latest log message from updatetz is

2020-10-22T23:03:54+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Oct 25 '20 05:10 apicht

Yeap, same for me. Hopefully it can be fixed soon

Oct 25 '20 06:10 maksimu

When I look at the logs of the cronjobber I see this errors that keep repeating over and over again:

2020-10-25T07:09:06.312819787Z cronjobber E1025 07:09:06.312568       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope
2020-10-25T07:09:07.182842095Z cronjobber E1025 07:09:07.182647       1 controller.go:122] can't list Jobs: jobs.batch is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "jobs" in API group "batch" at the cluster scope
2020-10-25T07:09:07.316420688Z cronjobber E1025 07:09:07.316221       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope

Maybe something is wrong w/ the service account? In fact my service account that was installed before was gone.

Oct 25 '20 07:10 maksimu

@maksimu Your errors should be unrelated. Your service account seems to be missing the correct ClusterRole permissions.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: cronjobber
rules:
- apiGroups: ['cronjobber.hidde.co']
  resources: ['*']
  verbs: ['*']
- apiGroups: ['batch']
  resources: ['jobs']
  verbs: ['*']
- apiGroups: ['']
  resources: ['events']
  verbs: ['create', 'patch', 'update']

Oct 25 '20 07:10 obitech

The problem is due to zic changing the default format from fat to slim since >=2020-b, this format is however not correctly understood by Go (https://github.com/golang/go/issues/42138), resulting in an UTC fallback.

The best option here is likely to start using Go 1.15, and make use of the embedded timezone information there, which makes the timezone update utility obsolete.

Oct 25 '20 11:10 hiddeco

@hiddeco do you find time to fix this? Otherwise I can draft a PR.

Oct 25 '20 11:10 obitech

I have just crafted a release, once it has been pushed to the registry, using the https://raw.githubusercontent.com/hiddeco/cronjobber/master/deploy/deploy.yaml manifest with quay.io/hiddeco/cronjobber:0.3.0 should solve this issue with a workaround.

I will track the Go issue to release a new version once it is able to understand the slim format as well, so the updatetz utility can be used again. Keeping this issue open until then.

Oct 25 '20 12:10 hiddeco

Works again with 0.3.0, thank you! 🙏

Oct 26 '20 15:10 obitech

The bug fix for this problem has landed in the release branch for Go 1.15.4, with the release scheduled for next week. Expect a patch release as soon as this version is available in container / image land. :sunflower:

Oct 29 '20 23:10 hiddeco

I am just wondering when cronjobber is going to be patched with the golang fix (starting from 1.15.4) for handling slim tzdata given that we are already past the 2021 DST transition. Thanks!

Mar 17 '21 18:03 rmak-cpi

Hello all,

It is likely not what you hoped, but I do at present not have the bandwidth to further maintain this project. See https://github.com/hiddeco/cronjobber/issues/43#issuecomment-890097573 for more information.

Jul 30 '21 19:07 hiddeco