[backend] Cannot set the explicit securityContext capabilities like SYS_NICE and privliged

[vivekrsintc]

Environment

• How did you deploy Kubeflow Pipelines (KFP)? Im using SDK it generates the spec where I could not set the security context privlige

https://www.kubeflow.org/docs/pipelines/installation/overview/. -->

• KFP version: kfp 2.14.3 kfp-pipeline-spec 2.14.0 kfp-server-api 2.14.0
• KFP SDK version: kfp 2.14.3 kfp-pipeline-spec 2.14.0 kfp-server-api 2.14.0

Steps to reproduce

Try to add securityContext options

Expected result

User should give same option as configure like k8s capabilities. at the end its going to deploy on kubernetes resource

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[Prateekbala]

Hi! I'd like to work on this issue. Could you assign it to me?

[juliusvonkohout]

Hi! I'd like to work on this issue. Could you assign it to me?

Done, but you might need help from @droctothorpe and @zazulam

[juliusvonkohout]

@droctothorpe @HumairAK @hbelmiro do you have any timeline for this feature? I think it is critical to sunset KFP v1.

We need a tight securitycontext such as

    securityContext:
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      seccompProfile:
        type: RuntimeDefault

[Prateekbala]

Hey @juliusvonkohout @droctothorpe

Before I start implementing, I wanted to clarify one thing - should the security context be applied at the container level, pod level, or should users be able to choose both?

Thanks!

[juliusvonkohout]

Hey @juliusvonkohout @droctothorpe

Before I start implementing, I wanted to clarify one thing - should the security context be applied at the container level, pod level, or should users be able to choose both?

Thanks!

Users should be able to set both. Some things can only be set at the pod level and some at the container level. so something like set_container_security_context() and set_pod_security_context(). set_container_security_context() is more important and the first one you should tackle.

The default for the container should be

    securityContext:
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      seccompProfile:
        type: RuntimeDefault

[HumairAK]

Can we clarify which user should be able to specify the SC in this case? This looks like an admin level concern, and not a pipeline author / data scientist concerns. We have several paths here with a possible hybrid approaches:

1. Just expose the ability for users to specify SC via kfp dsl for their pipelines via pipeline config, or possibly more granular to the component (KFP pipeline author concern)
2. Set the pod / container SC via KFP server deployment (KFP admin concern)
3. Allow both, with (1) allowing authors to override values in (2) if they are set on the pipeline - we expose all fields in both
4. Similar to (3), but we are opinionated about what you can set at KFP server level, and what we allow users to configure for a specific pipeline/component. For example we allow admin to set things like allowPrivilegeEscalation or privileged at server level but expose fields like runAsUser for authors

My current preference is to either go with (2) or (4). The reason being is that the Pipeline Runtime SA isn't tightly as scoped on permissions, because the Pod does wrapper code around the user execution code, so I am not comfortable allowing users to set SC on this pod via their workloads today. We will need further discussions on how to handle such a case.

cc @mprahl / @droctothorpe / @zazulam / @nsingla

[juliusvonkohout]

Can we clarify which user should be able to specify the SC in this case? This looks like an admin level concern, and not a pipeline author / data scientist concerns. We have several paths here with a possible hybrid approaches:

1. Just expose the ability for users to specify SC via kfp dsl for their pipelines via pipeline config, or possibly more granular to the component (KFP pipeline author concern)

Yes that is what we need as first step. It is a regression from V1 pipelines where this was possible. If a pipeline author writes a pipeline he should be able to set the securitycontext at least for the container in which the step runs. This can be then added as decorator to each pipeline step if needed. Or if you bring back the pipeline_configuration from V1 that is also an option. But some steps might need to run as a different non-rot user than others. So we need the per pipeline step setting as in V1.

2. Set the pod / container SC via KFP server deployment (KFP admin concern)

That already works via the argo workflow controller

3. Allow both, with (1) allowing authors to override values in (2) if they are set on the pipeline - we expose all fields in both

Yes the argo wrokflow controller only sets defaults that can be overwritten

4. Similar to (3), but we are opinionated about what you can set at KFP server level, and what we allow users to configure for a specific pipeline/component. For example we allow admin to set things like `allowPrivilegeEscalation` or `privileged` at server level but expose fields like `runAsUser` for authors

Yes, if its configurable also per pipeline step, not just the whole pipeline.

My current preference is to either go with (2) or (4). The reason being is that the Pipeline Runtime SA isn't tightly as scoped on permissions, because the Pod does wrapper code around the user execution code, so I am not comfortable allowing users to set SC on this pod via their workloads today. We will need further discussions on how to handle such a case.

Not giving access to the pod securitycontext, but just the container with the customer workload is fine for me.

cc @mprahl / @droctothorpe / @zazulam / @nsingla

Thank you for pushing this.