NeMo
NeMo copied to clipboard
Propagate PERF_ENV_VARS and overrides to SlurmExecutor container_env
Expose the container_env parameter of SlurmExecutor, explicitly set container_env for any environment variable being set. If you're setting performance variables for your run you expect them to be set in the container.
The pyxis/enroot defaults are to set all variables in the user environment in the container environment, except when that variable already exists in the container. Then it won't overwrite it unless '--container-env=VAR' is set.
Thinking on this a little more I think I should also update this to have a corresponding flag in argparse. Right now the only way I the user can get a var_name passed to container_env is by editing the source directly.