gke auth config-helper is hitting against too many open files

[cppforlife]

@andyshinn reported following error during kapp deploy.

Error: Listing schema.GroupVersionResource{Group:"certificates.k8s.io", Version:"v1beta1", Resource:"certificatesigningrequests"}, namespaced: false: Get https://x.x.x.x/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?labelSelector=kapp.k14s.io%2Fapp%3D1565298894854176000: error executing access token command "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=pipe: too many open files output= stderr=

it appears that number of allowed open fds is exceeded due to gke auth.

[andyshinn]

I think the problem is related to auth tokens expiring really fast for kubectl commands. It happened to work because I had a valid token from running a kubectl command before the deploy. according to https://github.com/helm/helm/issues/2249#issuecomment-294793552 the change was made way back in the Google Cloud SDK release 150 to use your own user credentials by default instead of application default credentials.

This works because kubectl commands run the credential helper to get new credentials. But commands directly to the cluster bypassing kubectl will not. The "too many files" part seems unrelated and I don't think this is a problem with the library or API.

The solution here is to set gcloud config set container/use_application_default_credentials true or export CLOUDSDK_CONTAINER_USE_APPLICATION_DEFAULT_CREDENTIALS=true in the shell (in case someone else finds this).

I think this can be closed unless you want to investigate more.

[cppforlife]

thanks for investigation 👍

[cppforlife]

it seems that few more people hit this problem. reopening to (a) improve error message with a note about above setting (b) investigate further if it could be avoided.

[XanderStrike]

I had this problem and, in addition to flipping the use application default credentials bit I actually had to specify export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json" as well. This is bizarre because I was able to create a cluster moments earlier without specifying them, but oh well. Hope this helps the next person to come along.

[erSitzt]

Im getting a similar error, not related to GKE This happens both with a Rancher created and an imported cluster. Both through the Rancher API-Proxy and with direct access to the K8S-Api

kapp delete -n kapp-apps -a portal-dummy-deployment
Target cluster 'https://k8s-zentrale-qa.mydomain.com' (nodes: k8s-zentrale-qa-master3, 7+)
11:39:35AM: debug: Resources: Ignoring group version: schema.GroupVersionResource{Group:"metrics.k8s.io", Version:"v1beta1", Resource:"pods"}
11:39:35AM: debug: Resources: Ignoring group version: schema.GroupVersionResource{Group:"coordination.k8s.io", Version:"v1beta1", Resource:"leases"}
11:39:35AM: debug: Resources: Ignoring group version: schema.GroupVersionResource{Group:"authentication.istio.io", Version:"v1alpha1", Resource:"policies"}
11:39:35AM: debug: Resources: Ignoring group version: schema.GroupVersionResource{Group:"rbac.istio.io", Version:"v1alpha1", Resource:"serviceroles"}
kapp: Error: Listing schema.GroupVersionResource{Group:"", Version:"v1", Resource:"secrets"}, namespaced: true:
  Get https://k8s-zentrale-qa.mydomain.com/api/v1/namespaces/cattle-prometheus-p-tl9qq/secrets?labelSelector=kapp.k14s.io%2Fapp%3D1599216767961971712: dial tcp 172.x.x.x:443: socket: too many open files

Btw. this one seems to be related to iTerm on MacOS, my colleague just found out ... I dont think its a problem of kapp and i didnt see an excessive use of FDs by kapp.