[BUG][OpenSearchDashboards] OpenSearch Dashboards fails to start when readOnlyRootFilesystem set to 'true'

[gsmith-sas]

Describe the bug When I set the readOnlyRootFilesystem key to 'true', the OpenSearch Dashboards pod cannot be started. The following messages appear in the pod log:

FATAL Error: Unable to write OpenSearch Dashboards UUID file, please check the uuid.server configuration value in opensearch_dashboards.yml and ensure OpenSearch Dashboards has sufficient permissions to read / write to this file. Error was: EROFS

In addition, I can find no example or documentation showing the configuration variable (uuid.server) mentioned in the error message.

To Reproduce Steps to reproduce the behavior:

1. I added the following stanza to my user-values.yaml file:

securityContext:
  readOnlyRootFilesystem: true

2. Deploy OpenSearch Dashboards using Helm (and pointing to the user-values.yaml file)
3. I can see the OpenSearch Dashboards pod starting up but fails fairly early with the error messages shown above.
4. If I remove that stanza, OpenSearch Dashboards comes up without problems.

Expected behavior I expected OpenSearch Dashboards to come up without problems. I would also expect to find some mention of the configuration variable (uuid.server) in the documentation or sample opensearch-dashboards.yml files.

Chart Name I've seen this error with using both OpenSearch Dashboards version 1.3.5 (Helm chart version 1.5.1) and 2.4.1 (Helm chart version 2.7.0).

Host/Environment (please complete the following information):

• Helm Version: 3.7.0
• Kubernetes Version: 1.22.3

Additional context

• I notice that readOnlyRootFilesystem key is included (on line 95) in the Helm charts values.yml file but commented out.
• I also noticed is that if I omit the readOnlyRootFilesystem key (which allows OpenSearch Dashboards to come up), and SSH onto the running OpenSearch Dashboard pod, the file /usr/share/opensearch-dashboards/data/uuid and owned by user "opensearch-dashboards" (UID:1000) rather than root.

[rishabh6788]

@prudhvigodithi Could you please advise here?

[gsmith-sas]

@rishabh6788 @prudhvigodithi I've just tested with OpenSearch Dashboards 2.6.0 and the problem persists. The log messages have not changed from what I reported earlier. Can either of you provide some guidance on how to overcome this and/or when it might be fixed? Thanks!

[gsmith-sas]

@rishabh6788 @prudhvigodithi

I may have found a work-around. Obviously, a true fix would be better but maybe this will work until then. I added an extra volume/mount for the uuid file to my yaml file and that appears to let OpenSearch Dashboards come up. Here's what I added to the yaml file:

 extraVolumes:
 - name: uuid-file
   emptyDir: {}
 
 extraVolumeMounts:
 - name: uuid-file
   mountPath: "/usr/share/opensearch-dashboards/data/"
   subPath: uuid

Based on a few minutes of quick testing, OSD appears to be functional at a basic level (documents are returned, security is enforced). Do you see any potential issues/problems with this work-around? As I said earlier, a real fix to eliminate the need for this kind of hack would be a much better solution. And I was not successful finding something similar for the OpenSearch readOnlyFileSystem (#369).

[prudhvigodithi]

Hey @gsmith-sas to work with readOnlyRootFilesystem you need a custom volume and mounts where the readOnlyRootFilesystem is not applicable. The solution you provided seem to be a right way to move forward. @DandyDeveloper @TheAlgo can you please advise? @bbarani Thank you