backup pod unable to start with failed to fetch service account token

[michaellee8]

Consistently got this error event when trying to startup a vitess cluster on GKE Autopilot running k8s v1.22.10 and vitess 14.0.1, tried both operator version 2.7.1 and 2.7.2 both failed. The cluster config I am using is similar to the exampledb as provided.

MountVolume.SetUp failed for volume "kube-api-access-x6hpj" : failed to fetch token: serviceaccounts "vitess-operator" is forbidden: the UID in the bound object reference (4d8d96fd-b40a-484d-90ad-a55e46df2f01) does not match the UID in record. The object might have been deleted and then recreated

I have found that the error messages comes from https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/noderestriction/admission.go#L537 so I am not sure if this is a operator problem or a k8s problem.

[michaellee8]

Some additional information:

I have tried v1.22 on minikube and it works, but v1.21 on GKE Autopilot stable and v1.22 on GKE Autopilot regular both got the exact same MountVolume.SetUp error. GKE Standard v1.20 works flawlessly through, will try to see if GKE Standard v1.22 works.

Edit: Tried GKE Standard v1.22 on Regular as well, working flawlessly, looks like a GKE Autopilot specific issue, will report it on Google Issue Tracker.

Edit: Opened a ticket on https://issuetracker.google.com/issues/242384329