Remote kubectl ignoring RBAC rule

[adrianAzoitei]

Summary

I have a microk8s single-node cluster running and would like to interact with it from a remote machine through kubectl. I have created a Role configured to only allow access to resources in the default namespace, and a RoleBinding to couple it to a User. So, following https://microk8s.io/docs/multi-user. The problem is, if I use the Static Token File authentication method, the user actually gets access to the entire cluster. I can, for example, list pods in the kube-system namespace. This is not an issue if I use x509 Client Certs for authentication.

kubectl get pod -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-7f9c69c78c-nfdzc                     1/1     Running   14         237d
calico-kube-controllers-f7868dd95-phnq9      1/1     Running   14         237d
dashboard-metrics-scraper-78d7698477-qpx5q   1/1     Running   14         237d
kubernetes-dashboard-85fd7f45cb-dlv4b        1/1     Running   14         237d
metrics-server-8bbfb4bdb-s6d7s               1/1     Running   14         237d
hostpath-provisioner-5c65fbdb4f-26f29        1/1     Running   14         237d
calico-node-6x96r                            1/1     Running   14         237d

What Should Happen Instead?

The RBAC rule should be respected with both kinds of authentication: Static Token File and x509 Client Certs. The rule is respected when using client certs.

kubectl get pod -n kube-system
Error from server (Forbidden): pods is forbidden: User "jenkins" cannot list resource "pods" in API group "" in the namespace "kube-system"

Reproduction Steps

1. Setup microk8s cluster.
2. Create Role.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: jenkins
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["deployments", "replicasets", "pods"]
  verbs: ["get", "list", "watch", "create"]

3. Create RoleBinding.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins-role-binding
subjects:
- kind: User
  name: jenkins
  apiGroup: ""
roleRef:
  kind: Role
  name: jenkins
  apiGroup: ""

4. Add a token entry to /var/snap/microk8s/current/credentials/known_tokens.csv with columns bearer_token,user,uid,"group1,group2,group3".
5. Add the user credentials to the microk8s config: microk8s kubectl config set-credentials jenkins --token=<BEARER_TOKEN>.
6. Fetch the client config with microk8s config.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: <HOST IP>
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: REDACTED
- name: jenkins
  user:
    client-certificate: <PATH_TO_CERT>/jenkins.crt
    client-key: <PATH_TO_KEY>/jenkins.key

7. On the remote, put the client config in ~/.kube/config, or another file and set the env. variable KUBECONFIG as the path to that file.
8. Set the contexts.context.user field in client config on the remote as the defined user (i.e. jenkins in this case).
9. Attempt to fetch pods from the kube-system namespace.

[ktsakalozos]

Hi @adrianAzoitei,

At some point you should call microk8s enable rbac. Out of the box RBAC is not enabled. Also after editing known_tokens.csv you should restart MicroK8s so the new token is loaded.

Here is what I got:

$ kubectl  get all -A
NAMESPACE     NAME                                             READY   STATUS    RESTARTS         AGE
kube-system   pod/dashboard-metrics-scraper-6b6f796c8d-kj5tm   1/1     Running   9 (3m13s ago)    11h
kube-system   pod/kubernetes-dashboard-765646474b-49hj6        1/1     Running   9 (3m13s ago)    11h
kube-system   pod/calico-kube-controllers-9c685ff74-pwdmm      1/1     Running   10 (3m13s ago)   12h
kube-system   pod/calico-node-n24nk                            1/1     Running   10 (3m13s ago)   12h
kube-system   pod/metrics-server-5f8f64cb86-rcsms              1/1     Running   12 (3m13s ago)   15h

NAMESPACE     NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
default       service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP    15h
kube-system   service/metrics-server              ClusterIP   10.152.183.251   <none>        443/TCP    15h
kube-system   service/kubernetes-dashboard        ClusterIP   10.152.183.16    <none>        443/TCP    11h
kube-system   service/dashboard-metrics-scraper   ClusterIP   10.152.183.218   <none>        8000/TCP   11h

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   15h

NAMESPACE     NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/metrics-server              1/1     1            1           15h
kube-system   deployment.apps/calico-kube-controllers     1/1     1            1           15h
kube-system   deployment.apps/dashboard-metrics-scraper   1/1     1            1           11h
kube-system   deployment.apps/kubernetes-dashboard        1/1     1            1           11h

NAMESPACE     NAME                                                   DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/metrics-server-5f8f64cb86              1         1         1       15h
kube-system   replicaset.apps/calico-kube-controllers-5db474cc7      0         0         0       15h
kube-system   replicaset.apps/calico-kube-controllers-9c685ff74      1         1         1       12h
kube-system   replicaset.apps/dashboard-metrics-scraper-6b6f796c8d   1         1         1       11h
kube-system   replicaset.apps/kubernetes-dashboard-765646474b        1         1         1       11h
$ microk8s.enable rbac
Infer repository core for addon rbac
Enabling RBAC
Reconfiguring apiserver
RBAC is enabled
$ kubectl  get all -A
Error from server (Forbidden): pods is forbidden: User "user" cannot list resource "pods" in API group "" at the cluster scope
Error from server (Forbidden): replicationcontrollers is forbidden: User "user" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
Error from server (Forbidden): services is forbidden: User "user" cannot list resource "services" in API group "" at the cluster scope
Error from server (Forbidden): daemonsets.apps is forbidden: User "user" cannot list resource "daemonsets" in API group "apps" at the cluster scope
Error from server (Forbidden): deployments.apps is forbidden: User "user" cannot list resource "deployments" in API group "apps" at the cluster scope
Error from server (Forbidden): replicasets.apps is forbidden: User "user" cannot list resource "replicasets" in API group "apps" at the cluster scope
Error from server (Forbidden): statefulsets.apps is forbidden: User "user" cannot list resource "statefulsets" in API group "apps" at the cluster scope
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "user" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" at the cluster scope
Error from server (Forbidden): cronjobs.batch is forbidden: User "user" cannot list resource "cronjobs" in API group "batch" at the cluster scope
Error from server (Forbidden): jobs.batch is forbidden: User "user" cannot list resource "jobs" in API group "batch" at the cluster scope

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.