opensearch-project
Work towards being able to run with securityContext restrictions
I have been working to get Opensearch to run on our OpenShift cluster that have a security policy that enforces certain things. This means that all containers needs to set a securityContext.
- Containers run with random UID
- Namespaces have resource quotas. This means that all containers needs to have a resource limit set
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
I have gotten as far as only allowing a single init container to run with read-write, while the rest of the opensearch containers run with read-only.
It would be great if the operator would enable the necessary changes to allow this when deploying opensearch using the operator as well.
These are the issues and solutions I have found so far
Helm
The docker image needs to be built with permissions that allow any user to read and execute the files inside the container
- I created a pull request for this, but no reply so far: https://github.com/opensearch-project/docker-images/pull/36
- This isn't really in the scope of the operator as anyone could build their own image and use with it
Disable the fsgroup-volume init container as it wanted to run as root
Exception in thread "main" java.nio.file.FileSystemException: /tmp/opensearch-13365302973015516133: Read-only file system
- Added a mount for /tmp to a emptyDir
Error opening log file 'logs/gc.log': Read-only file system
- Added a mount for /usr/share/opensearch/logs to a emptyDir
Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system
- Used the keystore container that reads a secret and generates the
opensearch.keystorefile and mounts it as a configmap into the opensearch container. This means that only the keystore container needs to be read-write.
Operator
The initHelper container cannot have a securityContext set
- Disabled the initHelper
Issues with setting securityContext on the bootstrap container.
- Turns out that settings
setVMMaxMapCounttotrueenabled a container that always runs as privileged. - Setting
setVMMaxMapCounttofalsesolved this
Hi @jbnjohnathan. If I read your list correctly there is nothing that must be implemented in the operator itself to make it run on OpenShift? Everything can be configured with the current featureset?
I have not enough knowledge about OpenShift to assess if all your changes/solutions are generic enough to have the operator do them automatically on OpenShift.
Would you be willing to submit a PR to document this in the userguide (ideally along with some yaml snippets for the needed configuration)?
One blocking issue found now: https://github.com/Opster/opensearch-k8s-operator/issues/613
Some of the mentioned issues are operator related.
Apart from the above linked issue it's also lack of possibility of mounting /tmp and /usr/share/opensearch/logs to an emptyDir
When it is possible to run opensearch container with the operator in read-only mode I can add the steps to the user guide.
Apart from the above linked issue it's also lack of possibility of mounting /tmp and /usr/share/opensearch/logs to an emptyDir
What is the problem/limitation there? The operator has support for emptyDir volumes via the additionalVolumes feature.
I have been working to get Opensearch to run on our OpenShift cluster that have a security policy that enforces certain things. This means that all containers needs to set a securityContext.
- Containers run with random UID
- Namespaces have resource quotas. This means that all containers needs to have a resource limit set
securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true seccompProfile: type: RuntimeDefaultI have gotten as far as only allowing a single init container to run with read-write, while the rest of the opensearch containers run with read-only.
It would be great if the operator would enable the necessary changes to allow this when deploying opensearch using the operator as well.
These are the issues and solutions I have found so far
Helm
The docker image needs to be built with permissions that allow any user to read and execute the files inside the container
- I created a pull request for this, but no reply so far: fix: correction to run as other user than opensearch docker-images#36
- This isn't really in the scope of the operator as anyone could build their own image and use with it
Disable the
fsgroup-volumeinit container as it wanted to run as rootException in thread "main" java.nio.file.FileSystemException: /tmp/opensearch-13365302973015516133: Read-only file system
- Added a mount for /tmp to a emptyDir
Error opening log file 'logs/gc.log': Read-only file system
- Added a mount for /usr/share/opensearch/logs to a emptyDir
Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system
- Used the keystore container that reads a secret and generates the
opensearch.keystorefile and mounts it as a configmap into the opensearch container. This means that only the keystore container needs to be read-write.Operator
The initHelper container cannot have a securityContext set
- Disabled the initHelper
Issues with setting securityContext on the bootstrap container.
- Turns out that settings
setVMMaxMapCounttotrueenabled a container that always runs as privileged.- Setting
setVMMaxMapCounttofalsesolved this
Hi @jbnjohnathan.
Can you please share what you did exactly in
Used the keystore container that reads a secret and generates the opensearch.keystore file and mounts it as a configmap into the opensearch container. This means that only the keystore container needs to be read-write.
Kind regards and thanks in advance.
Hello, Is there any progress in allowing the operator to run with the default SecurityContext and provide privileged escalation?
