Feature Reqest: Don't bind the admin.conf user to system:masters

[raesene]

edit by neolit123

tasks for 1.29

• [x] k/e tracking issue and KEP: https://github.com/kubernetes/enhancements/issues/4214 https://github.com/kubernetes/enhancements/pull/4218 https://github.com/kubernetes/enhancements/pull/4302
• [x] k/k changes: https://github.com/kubernetes/kubernetes/pull/121305 https://github.com/kubernetes/kubernetes/pull/121609 https://github.com/kubernetes/kubernetes/pull/121648 https://github.com/kubernetes/kubernetes/pull/121674 https://github.com/kubernetes/kubernetes/pull/121743 https://github.com/kubernetes/kubernetes/pull/121837 https://github.com/kubernetes/kubernetes/pull/121841
• [x] k/website changes: https://github.com/kubernetes/website/pull/43540 https://github.com/kubernetes/website/pull/43870
• [x] e2e test changes: https://github.com/kubernetes/kubeadm/pull/2949 https://github.com/kubernetes/kubeadm/pull/2951 https://github.com/kubernetes/kubeadm/pull/2952 https://github.com/kubernetes/test-infra/pull/31161 https://github.com/kubernetes/kubeadm/pull/2960

tasks for 1.30

• [x] https://github.com/kubernetes/kubernetes/pull/122786
  • cleanup of TODOs in https://github.com/kubernetes/kubernetes/pull/121305
  • cleanup of TODOs in https://github.com/kubernetes/kubernetes/pull/121841
• [x] update e2e:
  • https://github.com/kubernetes/kubeadm/pull/3012
  • https://github.com/kubernetes/kubeadm/pull/3015

---

original issue description

At the moment, when a new Kubeadm cluster is created, the default credential that is provided for cluster operators to use has a subject group of system:masters. e.g.

Subject: O = system:masters, CN = kubernetes-admin

This is a special group for which cluster-admin rights are hardcoded into the Kubernetes API server source code (https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/rbac/escalation_check.go#L38).

As a result, this credential cannot be revoked, if a cluster operator loses control of it, or if one of their admins moves team or leaves the organization, the only way to revoke access completely is to rotate the cluster certificate authority keys.

Whilst Kubernetes in general doesn't currently support certificate revocation (https://github.com/kubernetes/kubernetes/issues/18982) it could be possible to reduce the risk posed by a lost admin.conf file, by having a specific clusterrolebinding for the admin.conf user to the cluster-admin clusterrole.

This would allow cluster operators to largely revoke the rights assigned to that credential, by removing the binding. It would still have system:authenticated rights, but these are likely (in most clusters) to be significantly lower than cluster-admin.

[neolit123]

the admin.conf is like a root password...it must not be shared. given i've seen a few cases where users share the file, i'm going to create a PR to make this very clear in https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/.

it makes perfect sense for a deployer on cluster creation to sign break-glass credentials and that is what the admin.conf is.

As a result, this credential cannot be revoked, if a cluster operator loses control of it, or if one of their admins moves team or leaves the organization, the only way to revoke access completely is to rotate the cluster certificate authority keys.

sharing the admin.conf in such a scenario is not ideal. kubeadm gives you the kubeadm kubeconfig user command that can be used to create a kubeconfig for any user / group.

/kind documentation

[raesene]

With the lifetime on the certificate being a year, it seems likely that in many cases users who had valid access to the credential when the cluster was created, may no longer need access during that period.

I'd agree that admin.conf should be a break-glass credential, but it may benefit cluster operators to provide a mechanism for that credential to be revoked, given its lifetime.

[neolit123]

With the lifetime on the certificate being a year, it seems likely that in many cases users who had valid access to the credential when the cluster was created, may no longer need access during that period.

you can also rotate the cluster CA, which will invalidate the system:master credentials. https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/

I'd agree that admin.conf should be a break-glass credential, but it may benefit cluster operators to provide a mechanism for that credential to be revoked, given its lifetime.

that is a kubernetes problem, since revocation is not supported in core. https://github.com/kubernetes/kubernetes/issues/18982

something that we can improve with respect to kubeadm admin credentials is to sign two files:

• admin.conf: doesn't use system:masters but still has super powers
• superadmin.conf: uses system:masters and is the break-glass credential but this change needs a KEP / proposal and might not land anytime soon. i don't think i will have time for it in 1.22.

[neolit123]

/assign assigning myself to make it very clear what admin.conf is in https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

[neolit123]

@raesene please add LGTM to this PR: https://github.com/kubernetes/website/pull/27141

[raesene]

done :)

[mlbiam]

Would an off-line CA that could issue break-glass admin certs be more secure? you still have a "root" credential in the off-line CA that you would need to protect, but could issue shorter lived/time boxed admin.conf certs that could be controlled through other means.

[arianvp]

Would perhaps be useful to also put the suggestion of kubeadm kubeconfig user command in the output of kubeadm init as well so people really do not download it

[neolit123]

Would an off-line CA that could issue break-glass admin certs be more secure? you still have a "root" credential in the off-line CA that you would need to protect, but could issue shorter lived/time boxed admin.conf certs that could be controlled through other means.

by off-line do you mean a CA that the cluster components and kubeadm are not aware of, such as a root CA of the user org? if yes, i don't think signing the system:masters credentials with it would work. it must be signed with the intermediate CA that is passed to the cluster and if the credential leaks the same intermediate CA must be rotated (since, no revocation in k8s).

Would perhaps be useful to also put the suggestion of kubeadm kubeconfig user command in the output of kubeadm init as well so people really do not download it

there is a warning for that on this page: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Warning: Kubeadm signs the certificate in the admin.conf to have Subject: O = system:masters, CN = kubernetes-admin. system:masters is a break-glass, super user group that bypasses the authorization layer (e.g. RBAC). Do not share the admin.conf file with anyone and instead grant users custom permissions by generating them a kubeconfig file using the kubeadm kubeconfig user command.

but a message for that in the output of "init" seems fine for 1.23. PRs welcome. we are currently in code freeze for 1.22.

[arianvp]

Excellent. I'll draft a PR.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

/lifecycle frozen /unassign

[neolit123]

planning to draft a KEP for this in 1.29: https://github.com/kubernetes/enhancements/issues/4214

but it's low priority / might miss feature freeze for 1.29.

[pacoxu]

New TestGrid alerts at sig-cluster-lifecycle-kubeadm#kubeadm-kinder-super-admin-latest since commit/dbd3f3564.

--------------------------------------------------------------------------------
task-09-post-upgrade-primary-cp
Run commands after kubeadm upgrade is called on the primary CP node

command : /bin/bash -c set -x
CMD="docker exec kinder-super-admin-control-plane-1"

# Both admin.conf and super-admin.conf must exist
${CMD} test -f /etc/kubernetes/admin.conf || exit 1
${CMD} test -f /etc/kubernetes/super-admin.conf || exit 1

# Check certificate subject for .conf files
${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1
${CMD} grep 'client-certificate-data' /etc/kubernetes/super-admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = system:masters, CN = kubernetes-super-admin" || exit 1

# Check certificate subject for apiserver-kubelet-client.crt
${CMD} openssl x509 -subject -noout -in /etc/kubernetes/pki/apiserver-kubelet-client.crt | grep "subject=O = kubeadm:cluster-admins, CN = kube-apiserver-kubelet-client" || exit 1

# Check if the admin.conf user still has RBAC permissions
${CMD} kubectl -n kube-system --kubeconfig /etc/kubernetes/admin.conf get cm kubeadm-config || exit 1

# Ensure exit status of 0
exit 0

timeout : 5m0s
force   : false
--------------------------------------------------------------------------------

+ CMD='docker exec kinder-super-admin-control-plane-1'
+ docker exec kinder-super-admin-control-plane-1 test -f /etc/kubernetes/admin.conf
+ docker exec kinder-super-admin-control-plane-1 test -f /etc/kubernetes/super-admin.conf
+ exit 1

We may update the CI after we removed some logics in https://github.com/kubernetes/kubernetes/pull/122786/. Do I understand it correctly? @HirazawaUi @neolit123

[neolit123]

We may update the CI after we removed some logics in kubernetes/kubernetes#122786. Do I understand it correctly? @HirazawaUi @neolit123

yes, we need to update the test a little. i will send a PR.

[neolit123]

We may update the CI after we removed some logics in kubernetes/kubernetes#122786. Do I understand it correctly? @HirazawaUi @neolit123

yes, we need to update the test a little. i will send a PR.

PR https://github.com/kubernetes/kubeadm/pull/3012