Users can escalate privileges using Subscriptions

[chenzhiwei]

trafficstars

Bug Report

A user with CRUD permission on OperatorGroup/Subscription can escalate the cluster-admin permission, and then destroy the cluster if he want.

What did you do?

1. Create a namespace, service account and grant CRUD operatorgroup/subscription permissions to this service account

apiVersion: v1
kind: Namespace
metadata:
  name: test-ns

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: debug
  namespace: test-ns

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ns-admin
  namespace: test-ns
rules:
- apiGroups:
  - operators.coreos.com
  resources:
  - operatorgroups
  - subscriptions
  verbs:
  - '*'

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ns-admin
  namespace: test-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ns-admin
subjects:
- kind: ServiceAccount
  name: debug

2. Create a pod with the above service account

   We can see that the service account has no get pod permission, it actually only has CRUD on OperatorGroup and Subscription.

# kubectl -n test-ns run --generator=run-pod/v1 -it --rm --image=siji/kubectl --serviceaccount=debug debug
If you don't see a command prompt, try pressing enter.
bash-5.0# kubectl get pod
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:test-ns:debug" cannot list resource "pods" in API group "" in the namespace "test-ns"

3. Use the service account to install a cluster scoped operator

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-og
spec:
  targetNamespaces:
  - test-ns

---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: jenkins
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: jenkins-operator
  source: community-operators
  sourceNamespace: openshift-marketplace

4. The cluster scoped operator can be installed successfully

   I think this is a very serious privilege escalation issue.

What did you expect to see?

The user can't escalate any permissions.

Environment

• operator-lifecycle-manager version:

4.5.13 / 0.15.1

• Kubernetes version information:

Client Version: 4.4.6 Server Version: 4.5.13 Kubernetes Version: v1.18.3+47c0e71

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0-4-g38212b5", GitCommit:"4a4cd759b616cdba344dd73386727c10d3d2dde1", GitTreeState:"clean", BuildDate:"2020-05-25T22:39:54Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3+47c0e71", GitCommit:"47c0e71", GitTreeState:"clean", BuildDate:"2020-09-17T23:10:07Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

Possible Solution

Additional context Add any other context about the problem here.

[awgreene]

Good morning @chenzhiwei,

Thank you for using OLM and submitting this issue.

The OLM team views the installation of an operator as a highly privileged workflow that modifies the control plane of your cluster and recommends that this privilege be reserved for trusted Cluster Admins.

When using OLM to expand the control plane of your cluster, the OLM team recommends the following workflow:

• A Cluster Admin should review the operator that they wish to install.
• After deciding that the operator is safe and should be installed, the Cluster Admin should create a Subscription to install the operator.
• The Cluster Admin should then provide specific users with the necessary RBAC to create CR's that the operator watches.
• Users granted proper RBAC can now take advantage of the operator and its features.

[awgreene]

In the OLM Dev WG meeting, @njhale brought up that it is possible to specify a ServiceAccount's RBAC to install an Operator, preventing privilege escalation beyond the scope of the RBAC defined for the ServiceAccount.

To learn how to scope RBAC for operators installed in an OperatorGroup, please review this documentation.

[chenzhiwei]

@awgreene will user impersonation help on this?

OLM can act as the user who creates subscription to create the operater related resouces(service account, role, role binding, deployment, csv, installplan, etc).

https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation

[chenzhiwei]

Service Account can be also used for privilege escalation.

Caution: System administrators, use care when granting access to pod creation. A user granted permission to create pods (or controllers that create pods) in the namespace can: read all secrets in the namespace; read all config maps in the namespace; and impersonate any service account in the namespace and take any action the account could take. This applies regardless of authorization mode.

https://kubernetes.io/docs/reference/access-authn-authz/authorization/#privilege-escalation-via-pod-creation

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Aesylwinn]

Hello. I recently discovered this same behavior. This behavior is counter intuitive and unnecessary.

1. If I grant access for a user to do anything within a namespace, I would expect them to be scoped to that namespace. Unless something in that namespace had a cluster role and binding that granted access to cluster resources, or had been granted permissions to another namespace, I would not expect the user to be capable of gaining cluster admin access. With the abundance of CRDs and operators installed using OLM, assigning a blanket statement which grants access to all namespace scoped resources is more common than one may think. Having an intuitive permissions model is important, given the abundance of security breaches that are often the result of human error.

2. This is not consistent with how kubernetes handles (cluster) roles and role bindings. Even if a user has the ability to create a role binding, they cannot assign privileges beyond those they already have.

3. Only malicious users who do not fully declare their permissions would use and benefit from this behavior. In the above statements, @awgreene mentioned that creating or modifying namespace scoped operator groups and subscriptions is a set of permissions which only cluster admins should have. A cluster admin would already have permissions to do anything on a cluster, so they would be unaffected if OLM changed the behavior in such a way that prevented the creation of operator groups and subscriptions by unprivileged users using the user impersonation mechanism that @chenzhiwei linked.

[benruland]

To add a level of relevance for OpenShift installations:

OpenShift includes a ClusterRole aggregate-olm-edit that includes create permissions for Subscriptions and aggregates these to the admin and edit ClusterRole by default.

If now an OperatorGroup without annotated Serviceaccount exists in a namespace (which e.g. is the case if we install an arbitrary Operator to that namespace with the OpenShift Web Console), we do have the scenario described in this issue and users with associated edit or admin in this namespace are able to escalate their privileges.

[dmesser]

By default no one other than cluster-admin has the permission to create OperatorGroups. In a standard OpenShift deployment only one OperatorGroup instance exists which is the default cluster-wide one that the OpenShift console relies on for default installations (in openshift-operators). A regular namespace admin does not get permission to create OperatorGroups and thus cannot escalate privileges by installing privileged operators.

If an admin grants permissions OperatorGroup that equates to giving the user the ability to install any operator. That's why our recommendation to delegate install capability is for admins to pre-provision an OperatorGroup with a restricted ServiceAccount which then limits the permission scope that users can escalate to, to that of their own namespace.

[chenzhiwei]

@dmesser I don't think users are interested in your description about the OperatorGroup, what they care about is the fact and result.

Let's first introduce the two concepts: Namespace Admin and Cluster Admin.

• Namespace Admin means the Admin can CRUD any namespace scoped resources in a namespace.
• Cluster Admin means the Admin can do anything in the cluster.

As we can see that the Namespace Admin has fewer permissions than Cluster Admin, but the fact is the Namespace Admin can escalate the privileges and which is Namespace Admin == Cluster Admin.

I have another suggestion to fix this issue is to add a OperatorGroup namesapce whitelist to the OperatorGroup controller, let the controller only reconcile the OperatorGroup in whitelist namespaces.

[dmesser]

@chenzhiwei The description is vital to understand the issue. The simple answer is that OperatorGroup is a privileged API which by default only the Cluster Admin has access to. OperatorGroups that allow to install operators with escalating privileges are not supposed get added into tenant namespaces, so by default there is no Namespace Admin == Cluster Admin situation.

It only appears if you put an OperatorGroup into namespace a tenant has access to. OLM provides the require scoping controls on both the namespace level (spec.targetNamespaces) and the ServiceAccount Level (spec.serviceAccountName) to put OperatorGroup access into tenant namespace without causing privilege escalation.

But if you put an OperatorGroup without any of those safeguards into a tenant namespace, it's the same as installing an Operator in a tenant namespace and therefore exposing the privileged ServiceAccount to that tenant.

[benruland]

Thanks for taking a look into this @dmesser.

My takeaway is to not create OperatorGroups into tenant namespaces unless I perform additional scoping controls.

As an alternative I will prefer cluster-wide operators or, when requiring namespaced Operators, add them into a separate operator namespace, that is managing the tenant namespace using spec.targetNamespaces in the OperatorGroup.

With (OLM-2097) the create permissions from namespace admins for Subscriptions objects will hopefully be removed in the future, which would greatly reduce the chance of "accidential escalation" when an OperatorGroup exists in a tenant namespace.

[dmesser]

@benruland Agreed, when OLM-2097 lands even inadvertently placing the OperatorGroup will not allow for privilege escalation.

I should note that the new API design we are working on for OLM 1.x will not have this concept. The management of operators will be done with CRDs at the cluster scope and it will just be a single object, so giving permissions to tenants on this API will be the only way they could ever install an operator.