K8SSAND-104 ⁃ Add support for strict pod security policies

[mapster]

Is your feature request related to a problem? Please describe. In our organization we have pod security policies that, among other things, includes readOnlyRootFilesystem=true and explicitly sets ranges for runAsUser. The cassandra pods created through K8ssandra attempts to modify files in /etc/cassandra on the cassandra container, which is not allowed according to our PSPs. In addition cass-operator will not generate a securityContext for the cassandra pods, since CassandraDatacenter.Spec.DockerImageRunsAsCassandra is set to false in the k8ssandra template. This means that the pods will be mutated and assigned the minimum UID from the first allowed range in the PSP that applies to the cassandra pods, and if that UID happens to not be 999 the pods won't start since the user doesn't have write permissions to the files in /etc/cassandra.

Describe the solution you'd like There are many possible solutions, and some might fit better with the design principles of K8ssandra than others. Our suggestion is that K8ssandra allows overriding of the serviceAccountName for the cassandra pods (to allow users of k8ssandra to assign specific PSPs to cassandra pods created by k8ssandra), and that the cassandra pods be created with emptyDir volume mounts on /etc/cassandra for the cassandra container. The latter part will probably require changes to cass-operator as well.

Describe alternatives you've considered The only solution that we've come up with that will work without lowering the security requirements (by allowing write access to the root filesystem), is to implement a mutating webhook that modifies the cassandra pods with an emptyDir volume mount to /etc/cassandra, and switches to a custom cassandra image with an entrypoint that copies the /etc/cassandra files from the upstream image into the mounted volume at /etc/cassandra.

┆Issue is synchronized with this Jira Task by Unito ┆epic: Security Phase II ┆friendlyId: K8SSAND-104 ┆priority: Medium

[jsanda]

Our suggestion is that K8ssandra allows overriding of the serviceAccountName for the cassandra pods

This is an easy fix. It is just a matter of exposing another property in the CassandraDatacenter spec.

and that the cassandra pods be created with emptyDir volume mounts on /etc/cassandra for the cassandra container.

We actually already do this when backup/restore is enabled. The medusa container needs access to /etc/cassandra. We could use the emptyDir volume all the time, not just when backup/restore is enabled.

@mapster would you be interesting in submitting a PR for this? I am more than happy to assist you on it, and I may learn a bit more about PSPs 😄

[mapster]

Yesterday I also realized that we would need the cassandra containers to support running as non-root, e.g. the cassandra user. This PR seems to solve it, but it's been stagnant for several months now.

I'll try to submit a PR to you with the changes I've suggested.

[jsanda]

@emerkle826 do you think you could build new management-api images that include https://github.com/datastax/management-api-for-apache-cassandra/pull/43?

[jsanda]

I'll try to submit a PR to you with the changes I've suggested.

👍

Let me show you how we update the volumes of the CassandraDatacenter. Here is an example values.yaml file that configures backup/restore:

# values.yaml
backupRestore:
  medusa:
    enabled: true
    bucketName: my-s3-bucket
    bucketSecret: my-secret
    storage: s3

Then generate the CassandraDatacenter template:

$ helm template test ./k8ssandra -f test-values.yaml --show-only templates/cassandra/cassdc.yaml

The output should look like this:

---
# Source: k8ssandra/templates/cassandra/cassdc.yaml
apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
  labels:
    helm.sh/chart: k8ssandra-0.38.0
    app.kubernetes.io/name: k8ssandra
    app.kubernetes.io/instance: test
    app.kubernetes.io/version: "3.11.7"
    app.kubernetes.io/managed-by: Helm
  annotations:
    reaper.cassandra-reaper.io/instance: test-reaper-k8ssandra
spec:
  clusterName: k8ssandra
  serverType: cassandra
  serverVersion: "3.11.9"
  dockerImageRunsAsCassandra: false
  serverImage: datastax/cassandra-mgmtapi-3_11_9:v0.1.17
  managementApiAuth:
    insecure: {}
  size: 1
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
  allowMultipleNodesPerWorker: false
  users:
    - secretName: k8ssandra-reaper
      superuser: true
    - secretName: k8ssandra-medusa
      superuser: true
  config:
    cassandra-yaml:
      authenticator: PasswordAuthenticator
      authorizer: CassandraAuthorizer
      role_manager: CassandraRoleManager
      roles_validity_in_ms: 3.6e+06
      roles_update_interval_in_ms: 3.6e+06
      permissions_validity_in_ms: 3.6e+06
      permissions_update_interval_in_ms: 3.6e+06
      credentials_validity_in_ms: 3.6e+06
      credentials_update_interval_in_ms: 3.6e+06
    jvm-options:
      additional-jvm-opts:
        - "-Dcassandra.system_distributed_replication_dc_names=dc1"
        - "-Dcassandra.system_distributed_replication_per_dc=1"
  podTemplateSpec:
    spec:
      initContainers:

      - name: server-config-init
      - name: jmx-credentials
        image: busybox
        imagePullPolicy: IfNotPresent
        env:
          - name: JMX_USERNAME
            valueFrom:
              secretKeyRef:
                name: test-reaper-secret-k8ssandra
                key: username
          - name: JMX_PASSWORD
            valueFrom:
              secretKeyRef:
                name: test-reaper-secret-k8ssandra
                key: password
        args:
          - /bin/sh
          - -c
          - echo -n "$JMX_USERNAME $JMX_PASSWORD" > /config/jmxremote.password
        volumeMounts:
          - mountPath: /config
            name: server-config
      - name: get-jolokia
        image: busybox
        args:
          - /bin/sh
          - -c
          - wget https://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.2/jolokia-jvm-1.6.2-agent.jar && mv jolokia-jvm-1.6.2-agent.jar /config
        volumeMounts:
          - mountPath: /config
            name: server-config
      - name: medusa-restore
        image: k8ssandra/medusa:6ab6a55541e9
        imagePullPolicy: Always
        env:
          - name: MEDUSA_MODE
            value: RESTORE
          - name: CQL_USERNAME
            valueFrom:
              secretKeyRef:
                name: k8ssandra-medusa
                key: username
          - name: CQL_PASSWORD
            valueFrom:
              secretKeyRef:
                name: k8ssandra-medusa
                key: password
        volumeMounts:
          - name: test-medusa-config-k8ssandra
            mountPath: /etc/medusa/medusa.ini
            subPath: medusa.ini
          - name: server-config
            mountPath: /etc/cassandra
          - mountPath: /var/lib/cassandra
            name: server-data
          - name: medusa-bucket-key
            mountPath: /etc/medusa-secrets
      containers:
      - name: cassandra
        env:
          - name: LOCAL_JMX
            value: "no"
          - name: JVM_EXTRA_OPTS
            value: -javaagent:/etc/cassandra/jolokia-jvm-1.6.2-agent.jar=port=7373,host=localhost
        volumeMounts:
          - name: cassandra-config
            mountPath: /etc/cassandra
      - name: medusa
        image: k8ssandra/medusa:6ab6a55541e9
        imagePullPolicy: Always
        ports:
          - containerPort: 50051
        env:
          - name: MEDUSA_MODE
            value: GRPC
          - name: CQL_USERNAME
            valueFrom:
              secretKeyRef:
                name: k8ssandra-medusa
                key: username
          - name: CQL_PASSWORD
            valueFrom:
              secretKeyRef:
                name: k8ssandra-medusa
                key: password
        readinessProbe:
          exec:
            command: [ "/bin/grpc_health_probe", "-addr=:50051" ]
          initialDelaySeconds: 5
        livenessProbe:
          exec:
            command: [ "/bin/grpc_health_probe", "-addr=:50051" ]
          initialDelaySeconds: 10
        volumeMounts:
          - name: test-medusa-config-k8ssandra
            mountPath: /etc/medusa/medusa.ini
            subPath: medusa.ini
          - name: cassandra-config
            mountPath: /etc/cassandra
          - mountPath: /var/lib/cassandra
            name: server-data
          - mountPath: /etc/medusa-secrets
            name: my-secret
      volumes:
      - name: test-medusa-config-k8ssandra
        configMap:
          name: test-medusa-config-k8ssandra
          items:
            - key: medusa.ini
              path: medusa.ini
      - name: cassandra-config
        emptyDir: {}
      - name:  my-secret
        secret:
          secretName: my-secret

That is a lot, so let me highlight the relevant parts:

  spec:
# ...
    podTemplateSpec:
# ...    
      containers:
      - name: cassandra
        env:
          - name: LOCAL_JMX
            value: "no"
          - name: JVM_EXTRA_OPTS
            value: -javaagent:/etc/cassandra/jolokia-jvm-1.6.2-agent.jar=port=7373,host=localhost
        volumeMounts:
          - name: cassandra-config
            mountPath: /etc/cassandra
# ...
      volumes:
      - name: test-medusa-config-k8ssandra
        configMap:
          name: test-medusa-config-k8ssandra
          items:
            - key: medusa.ini
              path: medusa.ini
      - name: cassandra-config
        emptyDir: {}
      - name:  my-secret
        secret:
          secretName: my-secret

[mapster]

Thanks. I've gotten that part to work now. I tested it against our PSP, and found that container also needs write access to /opt/mcac-agent/config/metric-collector.yaml. Do you know if it is safe to mount an emptyDir to /opt/mcac-agent/config as well? There is one additional file in that path; collectd.conf.tmpl.

[jsanda]

Do you know if it is safe to mount an emptyDir to /opt/mcac-agent/config as well?

I think so.

[mapster]

If you're not sure, I could add an init container like so:

      - name: mcac-agent-config
        image: datastax/cassandra-mgmtapi-3_11_9:v0.1.17
        volumeMounts:
          - name: mcac-agent-config
            mountPath: /config
        command: ['/bin/sh']
        args: ['-c', 'cp -r /opt/mcac-agent/config /config']

[jsanda]

Let me clarify. I am confident it will work; however, I still want to cover my bases and test it to be certain.

[emerkle826]

@emerkle826 do you think you could build new management-api images that include datastax/management-api-for-apache-cassandra#43?

@jsanda @mapster Sorry I missed this, been heads down on the Management API stuff. I'm pretty confident I can get that PR merged with the 4.0-beta4 stuff. I hope to have it all done and released by tomorrow (Feb. 5 :crossed_fingers: )

[jsanda]

Thanks @emerkle826

@mapster I haven't forgotten about your PR :) I got tied up with some other issues and hope to circle back to it within the next couple days.

[mapster]

The logs of the server-system-logger container is filled with the following error messages when deploying with the changes I've made. I don't know if this actually is a problem yet, but I thought I'd let you know.

INFO  [ScheduledTasks:1] 2021-02-05 13:44:14,134 CollectdController.java:469 - /opt/mcac-agent/config/collectd.conf.tmpl
ERROR [ScheduledTasks:1] 2021-02-05 13:44:14,135 UnixSocketClient.java:380 - Error with collectd healthcheck
java.lang.RuntimeException: Unable to locate collectd.conf.tmpl
        at com.datastax.mcac.CollectdController$CollectdConfig.<init>(CollectdController.java:484) ~[datastax-mcac-agent.jar:na]
        at com.datastax.mcac.CollectdController$CollectdConfig.<init>(CollectdController.java:431) ~[datastax-mcac-agent.jar:na]
        at com.datastax.mcac.CollectdController$CollectdConfig$Builder.build(CollectdController.java:578) ~[datastax-mcac-agent.jar:na]
        at com.datastax.mcac.CollectdController.generateCollectdConf(CollectdController.java:235) ~[datastax-mcac-agent.jar:na]
        at com.datastax.mcac.CollectdController.start(CollectdController.java:286) ~[datastax-mcac-agent.jar:na]
        at com.datastax.mcac.UnixSocketClient.lambda$initCollectdHealthCheck$0(UnixSocketClient.java:368) ~[datastax-mcac-agent.jar:na]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-3.11.9.jar:3.11.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_282]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_282]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) ~[apache-cassandra-3.11.9.jar:3.11.9]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_282]
INFO  [insights-3-1] 2021-02-05 13:44:17,735 UnixSocketClient.java:789 - Metric reporting skipped due to connection to collectd not being established
WARN  [insights-3-1] 2021-02-05 13:44:17,738 NoSpamLogger.java:94 - Connection to Collectd not established

[mapster]

@emerkle826 I've rebased the changes onto master in https://github.com/datastax/management-api-for-apache-cassandra/pull/69

[jsanda]

Hey @mapster that stacktrace is coming from MCAC. It is a metric collector agent. I have not had a chance to look at it, but I am inclined to say it is a problem.

[mapster]

I'll test copying the files in /opt/mcac-agent/config from the container and into the emptyDir volume I've mounted there. e.g. something like:

      - name: mcac-agent-config
        image: datastax/cassandra-mgmtapi-3_11_9:v0.1.17
        volumeMounts:
          - name: mcac-agent-config
            mountPath: /config
        command: ['/bin/sh']
        args: ['-c', 'cp -r /opt/mcac-agent/config /config']

On another note, for k8ssandra to work with read only root fs and running as non-root users we also need some changes in Reaper. I'm going to look at that next week. (or perhaps in reaper-operator)

[emerkle826]

The Management API changes should be done now as of this PR being merged. All of the images should run Cassandra as the cassandra user for v0.1.20 and newer.

[mapster]

Ref: https://github.com/k8ssandra/k8ssandra/pull/301#issuecomment-777467013

Reaper still won't start with a PSP that has readOnlyRootFilesystem=true and nonRoot user. To fix it we need modifications to both k8ssandra/reaper-operator and thelastpickle/cassandra-reaper:

1. reaper-operator must add writeable volumes to reaper deployments
2. cassandra-reaper must change the location of /etc/cassandra-reaper.yml so that it is possible to mount a volume where the file is.

[mparikhcloudbeds]

Hi @jsanda , and others, I wonder if we would account for these in near future? : https://github.com/kubernetes/enhancements/pull/2582

[burmanm]

At some point, yes, given that PSP is deprecated.

[Jmoore1127]

This has been a large pain point during our evaluation/deployment of k8ssandra. Any movement on this issue (and related issues for subcharts like reaper)?

[jsanda]

Hi @Jmoore1127

Short answer... no movement

Longer answer... While PodSecurityPolicy is deprecated, I think it might be worthwhile to add support for it because k8ssandra will probably support kubernetes versions < 1.22 for some time. I personally don't have the cycles in the near term to work on this, but if you want to submit a PR, I would be happy to help with it and review as much as I can.

There is no subchart for Reaper. Are you referring to the reaper-operator subchart?

[jeffreyscarpenter]

Also @Jmoore1127 if it's easier/faster for you, feel free to chat on our Discord server https://discord.gg/qP5tAt6Uwt.

[Jmoore1127]

Thanks for a quick reply. Yes, I was referring to the operator subcharts. I think the main problem is that we enforce a read only filesystem in all our pods. Even as PSPs go away I expect we will still have the same security requirement, it will just be enforced by another admissions controller.

This has meant that getting the cassandra pods created by the DataCenter CR to successfully start up has been difficult. We have had to fork and make lots of modifications to the templates to mount emptyDir volumes so that directories such as /etc/cassandra and /opt/metrics-collector/config and even /tmp are writeable. This is undesireable since we can no longer easily update to the latest chart versions when they are released. It was surprising to us that it was not working out of the box given that the securityContext.readOnlyRootFilesystem exists and defaults true for many of the subcharts.

[jsanda]

@Jmoore1127 It does indeed look like we have some work to do. We already use an emptyDir volume for /etc/cassandra, but I believe volumes do need to be added for the other directories you mentioned.

While the subcharts declared readOnlyRootFilesystem: true it isn't actually set in the templates with the exception of the cass-operator chart.

I think changes are needed in the Reaper image as well.

In summary here is a list of changes that are needed:

• Make root file system read-only by default in reaper-operator chart
  • Properties are expose for the PodSecurityContext and container SecurityContext but are not actually used in the template
• Make root file system read-only by default in medusaa-operator chart
  • Properties are expose for the PodSecurityContext and container SecurityContext but are not actually used in the template
• Add emptyDir volume in Cassandra pod for /opt/metrics-collector/config
• Add emptyDir volume in Cassandra pod for /tmp
• Add support in reaper-operator for configuring PodSecurityContext and SecurityContext for containers
  • Make root file system read-only by default
• Make sure Reaper image is configured to run as non-root user
• Add emptyDir volume in Reaper pod for /etc/reaper

Did I miss anything?

Once we have this list firmed up I would like to create separate tickets for each of these items. We are targeting next week for a 1.3.0 release. These could definitely be done for the next release though.

[Jmoore1127]

I think that about sums it up for us. I think the only other (slight) roadbump is the awkwardness in having to enable medusa to get the /etc/cassandra volume to mount. We don't necessarily care about having medusa running in lower environments but still use a read only filesystem. It would be preferable for us if each component could be as self contained as possible.

[jsanda]

Ah, that's right. We're only using an emptyDir for /etc/cassandra when medusa is enabled. We want the emptyDir all the time. I'll add that to the list.

[jsanda]

@Jmoore1127

Do we need to explicitly declare a SecurityContext withreadOnlyRootFilesystem: true for each container? Previously I thought it could be configured in the PodSecurityContext, but it doesn't look like it.

cc @jeffbanks

[Jmoore1127]

Not for our use case. The read-only filesystem is enforced externally in our cluster (hence why the pods would fail to start when they couldn't write).

Perhaps there is an argument to be made for principle of least surprise here by setting it? Either way not critical for us.

[jeffbanks]

Thanks @Jmoore1127 for the follow-up information on this one!

[jeffbanks]

Summary of completed and pending work

Make root file system read-only by default in reaper-operator chart

• Merged: available as part of https://github.com/k8ssandra/k8ssandra/pull/1126

Make root file system read-only by default in medusa-operator chart

• Merged: available as part of https://github.com/k8ssandra/k8ssandra/pull/1126

Add support in reaper-operator for configuring PodSecurityContext and SecurityContext for containers

• Merged: available as part of PR: https://github.com/k8ssandra/k8ssandra/pull/1126

Make sure Reaper image is configured to run as non-root user

• Add emptyDir volume in Reaper pod for /etc/cassandra-reaper
• Merged as part of: https://github.com/k8ssandra/reaper-operator/pull/62 and https://github.com/k8ssandra/reaper-operator/pull/84

Summary of the default security settings in the K8ssandra security phase-1

• https://github.com/k8ssandra/k8ssandra/issues/1140

Tasks mentioned here and prioritized as part of K8ssandra security phase-2

Add emptyDir volume in Cassandra pod for /opt/metrics-collector/config

• Pending: work as part of https://github.com/k8ssandra/cass-operator/issues/212

Add emptyDir volume in Cassandra pod for /tmp

• Pending: work as part of https://github.com/k8ssandra/cass-operator/issues/199
• k8ssandra/cass-operator mention of /tmp-config-volume creation: https://github.com/k8ssandra/cass-operator/commit/0474057e8339da4f89b2e901ab697f10a2184d78

[adejanovski]

Closing as K8ssandra is now deprecated.