Reloader
Reloader copied to clipboard
stakater
[BUG] Reloader must crash on API issues
Describe the bug We have reloader issue recently due to a netpol issue on our side, but reloader looked fine pod side and we had no alerts. We missed that reloader was unable to talk to apiserver after startup and it broke some of our services relying on it.
To Reproduce Start reloader with no network permission to access to kube apiserver. It's green pod side
Expected behavior Pod must crash then we are alerted by kube state metrics alerts
Screenshots If applicable, add screenshots to help explain your problem.
Environment
- Operator Version: latest
- Kubernetes/OpenShift Version: 1.33
Additional context
@nerzhul i myself have not seen this practice in community. Do you have any examples which we can explore maybe? Also did you see any logs in the pod when it wasnt able to communicate with apiserver?
I have just seen this bug too.
Reloader logs were as follows:
time="2025-08-26T18:31:41Z" level=info msg="Environment: Kubernetes"
time="2025-08-26T18:31:41Z" level=info msg="Starting Reloader"
time="2025-08-26T18:31:41Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2025-08-26T18:31:41Z" level=info msg="created controller for: secrets"
time="2025-08-26T18:31:41Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2025-08-26T18:31:41Z" level=info msg="created controller for: configMaps"
time="2025-08-26T18:31:41Z" level=info msg="Starting Controller to watch resource type: configMaps"
E0829 10:45:40.557931 1 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: Failed to watch *v1.Secret: Get \"https://10.72.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=1756464315273976000&timeoutSeconds=435&watch=true\": http2: client connection lost" logger="UnhandledError"
It seems like the unhandled error, caused by temporary inability to connect to apiserver, has stopped reloader from watching for changes but didn't cause the liveness probe to fail and trigger a restart.
After restarting the pod manually, everything works normally.
