Kubernetes operator won't shut down cleanly, "unauthorized" errors

[hazeledmands]

What is the issue?

When I update my k8s config, the tailscale operator fails to shut down. Instead, I see the following messages in the logs for the operator, looping again and again:

E0828 06:21:38.885314       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: the server has asked for the client to provide credentials (get connectors.tailscale.com)
W0828 06:21:40.242623       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:21:40.242678       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:21:42.843275       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:21:42.843330       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:21:48.362002       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:21:48.362054       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:21:57.490310       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:21:57.490372       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:22:06.887965       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: the server has asked for the client to provide credentials (get ingresses.networking.k8s.io)
W0828 06:22:08.242885       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:22:08.242936       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:22:11.289046       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:22:11.289106       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:22:12.808682       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:22:12.808738       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:22:14.714764       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:22:14.714851       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
E0828 06:22:14.885038       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: the server has asked for the client to provide credentials (get serviceaccounts)
W0828 06:22:16.338063       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:22:16.338115       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:22:19.286735       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:22:19.286829       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:22:22.430246       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:22:22.430314       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:22:24.676987       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:22:24.677039       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:22:34.890444       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:22:34.890498       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:22:44.653579       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:22:44.653633       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:22:47.814906       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:22:47.814952       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:22:56.428429       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:22:56.428482       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:23:22.795034       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:23:22.795086       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:23:29.421143       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:23:29.421207       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:23:29.774355       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:23:29.774415       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:24:02.029609       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:24:02.029849       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized
W0828 06:24:02.598371       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:24:02.598423       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:24:02.986886       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:24:02.987094       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
E0828 06:24:29.889221       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: the server has asked for the client to provide credentials (get endpointslices.discovery.k8s.io)
W0828 06:24:30.856583       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: Unauthorized
E0828 06:24:30.856637       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
W0828 06:24:32.664970       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.ServiceAccount: Unauthorized
E0828 06:24:32.665021       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: Unauthorized
W0828 06:24:33.302897       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: Unauthorized
E0828 06:24:33.303089       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
W0828 06:24:33.973270       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Ingress: Unauthorized
E0828 06:24:33.973320       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Ingress: failed to list *v1.Ingress: Unauthorized
W0828 06:24:39.098942       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: Unauthorized
E0828 06:24:39.098994       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
W0828 06:24:45.725327       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: Unauthorized
E0828 06:24:45.725374       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
E0828 06:24:45.885316       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Deployment: the server has asked for the client to provide credentials (get deployments.apps)
W0828 06:24:47.172564       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Deployment: Unauthorized
E0828 06:24:47.172618       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Deployment: failed to list *v1.Deployment: Unauthorized
W0828 06:24:47.733919       1 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1alpha1.Connector: Unauthorized
E0828 06:24:47.733974       1 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1alpha1.Connector: failed to list *v1alpha1.Connector: Unauthorized

Steps to reproduce

• Install tailscale kubernetes operator (I did it using the helm chart at v1.72.1)
• Change something about the kubernetes configuration that causes it to terminate the operator, or just run kubectl delete -n tailscale <operator pod name>
• It hangs and logs look like the above

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

kubernetes version: v1.31.0

Tailscale version

tailscale kubernetes operator version 1.72.1

Other software

I have a pretty simple k8s stack:

• networking via flannel (barebones config from the flannel readme)
• prometheus-community/kube-prometheus-stack helm chart from https://prometheus-community.github.io/helm-charts
• tailscale/tailscale-operator helm chart from https://pkgs.tailscale.com/helmcharts

Bug report

BUG-060103701aa0998560314ae07e1692cb5444419ba5d91f85f27e34bbacd26452-20240828064032Z-51039b66bdd25a20