che When the disableContainerBuildCapabilities parameter is set to true, RoleBindings for Service Accounts are not created

Summary

Hello everyone,

I've been exploring running workspaces with custom container SecurityContexts and OpenShift SCCs. Here's what I've observed: when the disableContainerBuildCapabilities parameter is set to false, I noticed that in the user namespace, the operator creates RoleBindings to enable the use of the provided SCC by ServiceAccount.

However, when disableContainerBuildCapabilities is set to true, presumably due to the use of container SecurityContexts (which is why I need both SCC and SecurityContexts), I encountered a situation where the RoleBindings were not created.

I attempted to delve into the code for any clues but didn't find anything useful.

Could someone please help me understand if this is the expected behavior, or if there are additional configurations I need to provide? Any insights would be greatly appreciated.

Relevant information

Eclipse Che Version: 7.84.0 Cluster version: 4.15.0-0.okd-2024-03-10-010116

Below is the YAML configuration for Eclipse Che:

apiVersion: org.eclipse.che/v2
kind: CheCluster
metadata:
  annotations:
    che.eclipse.org/checluster-defaults-cleanup: '{"containers.resources":"true","spec.components.dashboard.headerMessage":"true","spec.components.pluginRegistry.openVSXURL":"true","spec.devEnvironments.defaultComponents":"true","spec.devEnvironments.defaultEditor":"true","spec.devEnvironments.disableContainerBuildCapabilities":"true"}'
  finalizers:
  - checluster.che.eclipse.org
  - cluster-resources.finalizers.che.eclipse.org
  - cheGateway.clusterpermissions.finalizers.che.eclipse.org
  - oauthclients.finalizers.che.eclipse.org
  - dashboard.clusterpermissions.finalizers.che.eclipse.org
  - consolelink.finalizers.che.eclipse.org
  - container-build.finalizers.che.eclipse.org
  name: eclipse-che
  namespace: openshift-operators
spec:
  components:
    cheServer:
      debug: false
      logLevel: INFO
    dashboard:
      logLevel: ERROR
    devWorkspace: {}
    devfileRegistry: {}
    imagePuller:
      enable: false
      spec: {}
    metrics:
      enable: true
    pluginRegistry: {}
  containerRegistry: {}
  devEnvironments:
    containerBuildConfiguration:
      openShiftSecurityContextConstraint: che-rootless-podman
    defaultNamespace:
      autoProvision: true
      template: <username>-che
    disableContainerBuildCapabilities: true
    maxNumberOfWorkspacesPerUser: 2
    secondsOfInactivityBeforeIdling: 1800
    secondsOfRunBeforeIdling: -1
    security:
      containerSecurityContext:
        allowPrivilegeEscalation: true
        capabilities:
          add:
          - SETGID
          - SETUID
          drop:
          - ALL
        readOnlyRootFilesystem: false
        runAsNonRoot: true
        seLinuxOptions:
          type: spc_t
    startTimeoutSeconds: 1000
    storage:
      pvcStrategy: per-user
  gitServices: {}
  networking:
    auth:
      gateway:
        configLabels:
          app: che
          component: che-gateway-config

When disableContainerBuildCapabilities is set to false, the RoleBindings are created correctly:

$ oc get rolebinding | grep che-
che-user-container-build                                     ClusterRole/che-user-container-build                                     18m
devworkspace-use-che-rootless-podman                         Role/devworkspace-use-che-rootless-podman                                17m

oc describe rolebinding/devworkspace-use-che-rootless-podman
Name:         devworkspace-use-che-rootless-podman
Labels:       app.kubernetes.io/name=devworkspace-workspaces
              app.kubernetes.io/part-of=devworkspace-operator
              controller.devfile.io/workspace-rbac=true
Annotations:  <none>
Role:
  Kind:  Role
  Name:  devworkspace-use-che-rootless-podman
Subjects:
  Kind            Name                          Namespace
  ----            ----                          ---------
  ServiceAccount  workspacec31b1c0c0e40428d-sa  rode-che-3p0eaa

When disableContainerBuildCapabilities is set to true, the RoleBindings are not created:

$ oc get rolebinding | grep che-
$ echo $?
1

Apr 25 '24 16:04 dmitryrode

Hello. This is expected, that when disableContainerBuildCapabilities: true [1] then roleBinding is not created. So, please use disableContainerBuildCapabilities: false.

[1] https://github.com/eclipse-che/che-operator/blob/a4e19ead9d324dce09f5ca400a5cb7eff3abd253/pkg/deploy/container-build/container_build.go#L44

Apr 29 '24 12:04 tolusha

Hi @tolusha, thank you sincerely for your response; your insights are immensely valuable to me.

Regarding the disableContainerBuildCapabilities parameter, it indeed appears prudent to set it to false to ensure the creation of the Service Account (SA). However, I'm curious about the functionality of custom containerSecurityContext. My understanding is that without specific bindings for the Service Account (SA), it might be challenging to implement a more tailored Security Context. From the docs

Disables the container build capabilities. When set to false (the default value), the devEnvironments.security.containerSecurityContext field is ignored, and the following container SecurityContext is applied: \n containerSecurityContext: allowPrivilegeEscalation: true capabilities: add: - SETGID - SETUID

Could you please shed more light on this? Don't you think there's room for improvement here?

May 01 '24 10:05 dmitryrode

Hello @dmitryrode

If you need to use the custom SCC, then:

Create a custom SecurityContextConstraints object.
Set the name of the SecurityContextConstraints object into CheCluster CR.

With regards to the disableContainerBuildCapabilities field. It seems the description confuses. Correction:

When set to **true**, the the devEnvironments.security.containerSecurityContext field is ignored ...

May 02 '24 09:05 tolusha

To be honest I don't like this snippet, for me it looks inconsistent [1] @AObuchow Could you have a look please? Do we really want to set default security context when build capabilities are enabled?

https://github.com/eclipse-che/che-operator/blob/a4e19ead9d324dce09f5ca400a5cb7eff3abd253/pkg/deploy/dev-workspace-config/dev_workspace_config.go#L123-L140

May 02 '24 09:05 tolusha

Hello @tolusha, thank you for sharing the details. I attempted to adjust the parameters, but integrating them with OpenShift appears somewhat convoluted. But I can confirm custom SCC works fine :) But only whendisableContainerBuildCapabilities = true.

Let me elaborate further details:

Initially, I created a new custom SCC to grant Service Accounts (SA) more relaxed capabilities than the defaults.
Consequently, upon creating workspaces, everything appeared to function smoothly. However, this was only with the default containerSecurityContext in workspace Deployments, limited to SETGID and SETUID permissions [1].
Yet, imho it seems redundant to establish a custom SCC without providing corresponding modifications to the Container Security Context, such as capabilities adjustments or SELinux tweaks. In this case I need provide my capabilities and SELinux and disable default build capabilities. Then go to the 4th point.
Enabling custom Container Security Context necessitates setting disableContainerBuildCapabilities = true.
After the point 4th, SA creation performed without RBAC (RoleBinding to the Role allowing usage of securityContextConstraints).

Does it genuinely make sense to proceed in this manner in case of using custom Security Contexts?

I believe you are right and the snippet from the comment above looks inconsistent :) Otherwise it turns out that we are losing the ability to manage SCC and Container Security in such configurations.

[1] https://github.com/eclipse-che/che-operator/blob/main/pkg/deploy/container-build/container_build.go#L273

May 02 '24 10:05 dmitryrode

@tolusha, I mean the following:

the former pathway appears functional:

Custom OpenShiftSecurityContextConstraint → Workspace Creation → Automatic SA Generation → Automatic RBAC Establishment (RoleBinding with permissions to utilize your SCC) → Workspace Execution.

However, the pathway involving container security context encounters issues:

Custom OpenShiftSecurityContextConstraint → Custom ContainerSecurityContext → Workspace Creation → Automatic SA Generation → Absence of RBAC Setup → Workspace Failure due to SA inability to utilize the custom OpenShift SCC.

May 02 '24 10:05 dmitryrode

Hello @dmitryrode

I see. Removing the check [1] should allow to configure default container security context along with custom scc. The reason, why it has been done in that way is [2] The author @AObuchow is on PTO know. Once he is available, I will contact him. But personally, I don't see any problem with removing that if clause.

[1] https://github.com/eclipse-che/che-operator/blob/a4e19ead9d324dce09f5ca400a5cb7eff3abd253/pkg/deploy/dev-workspace-config/dev_workspace_config.go#L125 [2] https://github.com/eclipse-che/che/issues/22307

The workaround is to update the default container security context. So, please update Eclipse Che subscription in this way:

  config:
    env:
      - name: CHE_DEFAULT_SPEC_DEVENVIRONMENTS_CONTAINERSECURITYCONTEXT
        value: '<JSON_STRING_REPRESENTING_CONTAINER_SECURITY_CONTEXT>'

For instance:

  config:
    env:
      - name: CHE_DEFAULT_SPEC_DEVENVIRONMENTS_CONTAINERSECURITYCONTEXT
        value: '{"allowPrivilegeEscalation": true,"capabilities": {"add": ["SETGID", "SETUID"]}}'

May 06 '24 11:05 tolusha

Hello @tolusha, thank you very much for the provided insights. I'll be sure to give the proposed workaround a try.

Looking forward to hearing @AObuchow's feedback as well.

May 06 '24 11:05 dmitryrode

@dmitryrode @tolusha IIRC, we don't allow modifying the containerSecurityContext when container builds are enabled, as a way to ensure the correct container-build SCC is selected to permit building containers in a workspace. If a workspace service account has multiple SCCs available, a suitable SCC that has the least amount of privileges is picked. See the original issue description.

In other words, we intended to always have the container security context set to the following to ensure the provided container builds SCC would work:

	  containerSecurityContext:
	    allowPrivilegeEscalation: true
	    capabilities:
	      add:
	      - SETGID
	      - SETUID

I do think this restriction could be removed, however. If a user is confident in configuring their own SCC as well as the pod & container security contexts, we should assume that they are aware that they might risk breaking workspaces that rely on the container build SCC.

While working on allowing users to configure their workspace pod and container security contexts, I actually vaguely remember trying to do a strategic merge in the case the admin modifies the devEnvironments.security.containerSecurityContext Che Cluster CR field while container builds capabilities are enabled. The idea was to ensure allowPrivilegeEscalation is always set to true and the SETGID, SETUID capabilities are added and never dropped. Unfortunately, this idea was dropped at some point while working on the PR, and the changes were lost.

May 10 '24 21:05 AObuchow