Storage permission issues in IKS

[iantonov01]

Hello, I am trying to deploy IBM MQ using the last ibm-mqadvanced-server-dev helm chart in IKS but I am having a problem with the permissions of the files and folder created on the shared block storage.

I have applied this DaemonSet on the cluster to enable no_root_squash which I think is required for the deployment to work https://github.com/IBM-Cloud/kube-samples/blob/master/daemonset-sample/norootsquash.yaml

The values.yaml file I am using:

license: accept
image:
  tag: latest
version: 6.0.0
queueManager:
  name: TEST001
  multiInstance: True
  dev:
    secret:
      name: mq-secret
      adminPasswordKey: adminPassword
security:
  initVolumeAsRoot: yes
dataPVC:
  storageClassName: ibmc-file-gold-ibmmq
logPVC:
  storageClassName: ibmc-file-gold-ibmmq
qmPVC:
  storageClassName: ibmc-file-gold-ibmmq

After running the helm deployment the pods end up in crash loop. Those are their logs:

2021-11-17T20:22:20.297Z CPU architecture: amd64
2021-11-17T20:22:20.297Z Linux kernel version: 4.15.0-132-generic
2021-11-17T20:22:20.297Z Container runtime: kube
2021-11-17T20:22:20.298Z Base image: Red Hat Enterprise Linux 8.4 (Ootpa)
2021-11-17T20:22:20.298Z Running as user ID 1001 with primary group 0
2021-11-17T20:22:20.298Z Capabilities: none
2021-11-17T20:22:20.310Z seccomp enforcing mode: disabled
2021-11-17T20:22:20.310Z Process security attributes: cri-containerd.apparmor.d (enforce)
2021-11-17T20:22:20.311Z Detected 'nfs4' volume mounted to /mnt/mqm-log
2021-11-17T20:22:20.311Z Detected 'nfs4' volume mounted to /mnt/mqm-data
2021-11-17T20:22:20.311Z Detected 'nfs4' volume mounted to /mnt/mqm
2021-11-17T20:22:20.311Z Multi-instance queue manager: enabled
2021-11-17T20:22:20.774Z Using queue manager name: TEST001
2021-11-17T20:22:20.846Z Created directory structure under /var/mqm
2021-11-17T20:22:20.846Z Image created: 2021-07-09T06:03:07+00:00
2021-11-17T20:22:20.847Z Image tag: ibm-mqadvanced-server-dev:9.2.3.0-r1.20210709055735.f5fdd10-amd64
2021-11-17T20:22:20.920Z MQ version: 9.2.3.0
2021-11-17T20:22:20.920Z MQ level: p923-L210708.DE
2021-11-17T20:22:20.920Z MQ license: Developer
2021-11-17T20:22:25.049Z Creating queue manager TEST001
2021-11-17T20:22:25.049Z Starting web server
2021-11-17T20:22:25.470Z Error 2 creating queue manager: AMQ6239E: Permission denied attempting to access filesystem location
'/mnt/mqm-data/qmgrs/TEST001'.
AMQ7002E: An error occurred manipulating a file.

2021-11-17T20:22:25.470Z /opt/mqm/bin/crtmqm: exit status 2

Those are the files and folders created on the shared storage:

# ls -la
total 64
drwxrwsr-x   13 root     root          4096 Nov 17 20:00 .
drwxr-xr-x    3 root     root          4096 Nov 17 20:00 ..
drwxrwsr-x    2 root     root          4096 Nov 17 20:00 config
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 conv
drwxrwsrwx    2 root     root          4096 Nov 17 20:22 errors
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 exits
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 exits64
drwxrwsr-x    2 root     root          4096 Nov 17 20:00 log
-rw-rw-r--    1 root     root           671 Nov 17 20:00 mqclient.ini
drwxrwsr-x    5 root     root          4096 Nov 17 20:00 mqft
-rw-rw-r--    1 1001     root          1288 Nov 17 20:00 mqs.ini
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 qmgrs
-rw-rw-r--    1 root     root          1941 Nov 17 20:00 service.env
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 sockets
drwxrwsrwx    2 root     root          4096 Nov 17 20:00 trace
drwxrwsr-x    3 root     root          4096 Nov 17 20:00 web

From what I have read while trying to debug the issue, looks like that the files and folders probably do not have the expected permissions.

I have managed to make the deployment work by adding the init container below but I would expect that this step should not be needed if the deployment worked as expected.

      initContainers:
      - args:
        - chown 1001:0 /mnt -R;
        command:
        - /bin/sh
        - -c
        image: alpine:latest
        imagePullPolicy: Always
        name: fs-init
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/mqm
          name: data
        - mountPath: /mnt/mqm-log
          name: ibm-mq-ibm-mq-log
        - mountPath: /mnt/mqm-data
          name: ibm-mq-ibm-mq-qm

I would be grateful to hear your thoughts on the issue.

[arthurbarr]

The Helm charts are no longer supported, and haven't been tested with recent versions of MQ. But I'll see if I can help anyway.

The ibmc-file-gold StorageClass requires you to run the container as root. You can do this with the old Helm charts by setting the parameter initVolumeAsRoot to true. This will run an init container as root, set the permissions on the files correctly, then run the main container as a less privileged user.

Alternatively, the ibmc-file-gold-gid StorageClass just requires the container user to be in the root group (GID 0). Container versions from MQ 9.1.5 onwards are all set up to assume that the container user is in the root group (which has no other special security significance, unlike the root user). When you use this approach, no init container is needed — that makes this option the most secure approach.

You shouldn't have to use a custom StorageClass or do anything special for root squashing, AFAIK. Using ibmc-file-gold-gid is probably the best approach, I think.