signac-flow icon indicating copy to clipboard operation
signac-flow copied to clipboard

Support modification of the subprocess environment created when groups fork to execute operations

Open vyasr opened this issue 3 years ago • 1 comments

Feature description

Operations that require forking, either via an explicit fork directive or using any other directives that trigger forking like nranks (via MPI) or executable, end up executing operations in a subprocess of the process where the job script and the original signac-flow launch occur. Our Jinja templates offer various methods for modifying the parent process, and users can also write custom templates to modify the environment however they need. However, we don't currently offer any method for controlling the child processes. It would be nice to include some way of doing so.

Proposed solution

Currently the best solution I was able to suggest was to use the executable directive to just include a compound command, something like DO_THIS && DO_THIS && NOW_RUN_ORIGINAL_COMMAND. Maybe something like this is the best that we can do; if so we should document it as such in places like here.

vyasr avatar May 25 '21 19:05 vyasr

We could allow for a setup script for any forked commands to ensure that the modules are loaded as necessary. Otherwise we could document using the correct configuration files on a user shell to load necessary modules.

This could be like a template effectively except there would be no default.

Edit: Elaborated.

b-butler avatar May 25 '21 20:05 b-butler