More granular `securityContext` for Executor Containers

[agilgur5]

Summary

The Executor Pod has multiple different containers (init, wait, etc), some of which require different permissions (securityContext) than others. Depending on the type of Execution (e.g. resource), the required permissions can be different as well

Details

Most of this seems to come after the lock down of resource pods in #8657.

That feature changes the default securityContext for resource pods and also adds a carve out for the init and wait container of a resource executor, as those cannot have a readOnlyRootFilesystem.

But, one can only configure a single securityContext for the entire Executor. This means that if a user wants to use the resource type, the operator must allow readOnlyRootFilesystem: false in their executor.securityContext.

Example

I recently noticed that the initContainer of a resource requires readOnlyRootFilesystem: false when I had an Executor Pod fail to start-up (and therefore fail to report any logs etc). It had an Init:Error like the below:

❯ kubectl -n agilgur5-test get pod
NAME                                  READY   STATUS       RESTARTS   AGE
agilgur5-test-wft-77gkn-2301407197    0/1     Init:Error   0          29m
agilgur5-test-wft-ls8q8-2301407197    0/1     Init:Error   0          18m

And checking the status of the Pod, the initContainer had an error: message: 'open /tmp/manifest.yaml: read-only file system'.

Use Cases

1. Ability to lock down different containers and resource types of the Executor
2. Least privilege permissions per container and type
3. Not giving overly broad permissions due to "lowest common denominator" of Executor requirements, instead of more granular permissions

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[agilgur5]

As far as I can tell, there's also no way of mounting an emptyDir (or any volume) onto the Executor initContainer. So, in general, more granular config around the different Executor Containers could allow for attaching an emptyDir to /tmp for the initContainer which may avoid this error as well

[agilgur5]

podSpecPatch may be a workaround for this. Got mentioned in Slack recently

[jackivanov]

Can anyone share an example please?

adding workflowDefaults.spec.podSpecPatch to the config map does nothing to the init container

workflowDefaults:
  spec:
    podSpecPatch: |
      initContainers:
        - name: init
          securityContext:
            capabilities:
              drop:
              - ALL

[agilgur5]

podSpecPatch does work per https://github.com/argoproj/argo-workflows/issues/10787#issuecomment-2056664894. Not sure about in workflowDefaults.

But dropping all capabilities is also the default for a resource template (which is the main one I wrote about it in the issue description), so there would be no change for your patch