loading stale kubeconfig for GKE will result in error

[edaniszewski]

if using a GKE deployment that has been sitting for a while, then running a test using kubetest, you may get:

INTERNALERROR>     if self._load_auth_provider_token():
INTERNALERROR>   File "/Users/edaniszewski/dev/vaporio/kubetest/.tox/py36/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 196, in _load_auth_provider_token
INTERNALERROR>     return self._load_gcp_token(provider)
INTERNALERROR>   File "/Users/edaniszewski/dev/vaporio/kubetest/.tox/py36/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 236, in _load_gcp_token
INTERNALERROR>     self._refresh_gcp_token()
INTERNALERROR>   File "/Users/edaniszewski/dev/vaporio/kubetest/.tox/py36/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 245, in _refresh_gcp_token
INTERNALERROR>     credentials = self._get_google_credentials()
INTERNALERROR>   File "/Users/edaniszewski/dev/vaporio/kubetest/.tox/py36/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 139, in _refresh_credentials
INTERNALERROR>     scopes=['https://www.googleapis.com/auth/cloud-platform']
INTERNALERROR>   File "/Users/edaniszewski/dev/vaporio/kubetest/.tox/py36/lib/python3.6/site-packages/google/auth/_default.py", line 306, in default
INTERNALERROR>     raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
INTERNALERROR> google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://developers.google.com/accounts/docs/application-default-credentials.

this is fixed by running any kubectl command, presumably to refresh the token, but that's pretty hacky. we'll need to look into refreshing automatically if possible.

[hoanhan101]

Hmm I also got into this recently. For one time, I only needed to run a single kubectl command but other times, I had to run multiple ones and waited for a while.

[edaniszewski]

Yeah, the behavior is weird. Usually, running a single command seems to refresh the credentials, but not always, in which case I usually have to wait until they expire again, then run another command to refresh and then it usually works.

I started something to try and fix this, but I ended up doing something weird to my credentials which required me to delete and recreate my cluster. I'm hoping to look into this more today/tomorrow.

[edaniszewski]

this is still an issue. the workaround still works, even if its inconvenient. I did put some stuff together which I think should fix it but that gave me some trouble. I think that was related to the permissions on the service account I made, so it may just be a matter of getting the right permissions there.