LLNL
Document a fallback for CMA (cross memory attach)
Mercury uses CMA for shared memory communication. CMA is not enabled on all systems, and when it is, there are some cases where the OS disables it at runtime.
Even if CMA is enabled for users in general, the OS can disable it in certain situations. For example, if the read bit is not set on the executable, I think CMA is disabled. We have run into this where some binaries have their execute bit set but not their read bit. That enables someone to run a code but not read any of its bits. I wonder if we might see the same thing if we disable the read bit on the UnifyFS client application.
As Phil Carns noted in his email, mercury 2.x has an explicit check for this that will tell users if CMA was detected as supported by the configuration but not runtime enabled. Another reason we should try to move to newer margo/mercury/argobots versions.
We should look for and document fallbacks for those cases where it's needed. If Mercury can switch to use standard shared memory instead, that'd be a good option. Alternatively, we could modify UnifyFS to let the user set up a socket connection between the client and server.
FYI, Mercury 2.x is intended to at least print a friendly error message in the case where it detected that a) the platform supports CMA but b) due to security/permission settings it didn't actually work at run time. See https://github.com/mercury-hpc/mercury/blob/master/src/na/na_sm.c#L4680.
Unfortunately that error message was disabled by default in an effort to generally make Mercury less noisy, making it a little easier to hit this problem without Mercury telling you why it can't communicate. There is an issue tracking to track that (the need to audit which messages are on and off by default) at https://github.com/mercury-hpc/mercury/issues/426 .
That doesn't change the core problem in this issue, just a heads up.
