Kubernetes 1.21+: Token-handling code assumes auto-created service account token secrets

[liggitt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Sweeping token-scraping of auto-generated Kubernetes token secrets in preparation for Kubernetes 1.24 showed the following code locations assume auto-generated tokens will exist:

https://github.com/hashicorp/consul-k8s/blob/7ae81732981bb971243b52e4b25f08e80f5ccfd7/control-plane/connect-inject/handler.go#L279-L283

https://github.com/hashicorp/consul-k8s/blob/7ae81732981bb971243b52e4b25f08e80f5ccfd7/acceptance/framework/vault/vault_cluster.go#L173-L180

That assumption is not universally correct.

In 1.21+, secret-based tokens are no longer used for mounting into pods (ephemeral time-limited tokens are), and the token controller can be turned off.

In 1.24+, secret-based tokens are no longer auto-created by default for new service accounts.

Reproduction Steps

Create a 1.24 Kubernetes cluster or a 1.21+ Kubernetes cluster with the token controller disabled.

Additional Context

Using ephemeral time-bound tokens is preferred in 1.21+ (see the TokenRequest API) if possible.

If a secret-based token is still desired, one can be created manually, but will not be referenced from the service account's .secrets list.

[david-yu]

Thank you @liggitt for pointing out the API changes that will be landing in 1.24 shortly. We'll review the KEP as well to see how we remediate the API compatibility issue that will surface in 1.24: https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/2799-reduction-of-secret-based-service-account-token

Out of curiosity, how did you find out that we were using auto-generated Kubernetes token secrets?

[liggitt]

https://grep.app/search?q=.Secrets%5B0%5D.Name&filter[lang][0]=Go&filter[repo.pattern][0]=hashicorp

[vaibhavgulati]

Its been 4months since it got reported, when are we planning to close this?

[david-yu]

Hi @vaibhavgulati we are actively working on closing this issue out. Out of curiosity what distro and version of K8s are you using?

[dagtveit]

i am experiancing the same issue when trying to do a fresh install from helm chart with manageSystemACLs: true- deleted pvc and secrets multiple time. i am on 1.24.3 now. not shure what distro it is. but this is a scaleway managed cluster

[david-yu]

Thanks for the heads up @dagtveit. We are targeting early next month at the latest to support K8s 1.24.

[dagtveit]

Early next what. ? 1.24 was just default on the cluster i setup so i can try lower. what is the highest supported version then ?

[david-yu]

We will be planning on releasing support for 1.24 early next month, you should use 1.23.x and lower until then.

[vaibhavgulati]

@david-yu I am using eks 1.22 and it is not working there as well

[vaibhavgulati]

@david-yu we are way too behind in the release of this bug 1.21+ the way kubernetes version is progressing people will loose hope over it. Can we have a slack channel to actively track this?

[david-yu]

We have a PR up to address this but it still needs to reviewed, merged and released: https://github.com/hashicorp/consul-k8s/pull/1431. Stay tuned.

[vaibhavgulati]

@kschoche can you please review and merge?

[vaibhavgulati]

@david-yu this seems closed right?