smartdispatch icon indicating copy to clipboard operation
smartdispatch copied to clipboard

Don't export the user's environment to the job

Open bartvm opened this issue 8 years ago • 9 comments

Fixes #157.

Currently the user's entire environment is exported. This is bad practice because it makes jobs hard to reproduce and debug, and in some cases it can even cause errors. The only variable that needs to be exported is PBS_FILENAME, which is used by the auto-resume script.

bartvm avatar Feb 07 '17 19:02 bartvm

Coverage Status

Coverage remained the same at 94.532% when pulling 2042df349319eb0ebff3ba8e3dda0e5a35d4723b on bartvm:master into c94f08e828d6deec453c054fd340e694ec0e0b9a on SMART-Lab:master.

coveralls avatar Feb 10 '17 20:02 coveralls

@mgermain Test away.

bartvm avatar Feb 10 '17 20:02 bartvm

@mgermain Ping.

bartvm avatar Feb 17 '17 18:02 bartvm

@bartvm working on one cluster, not working on the other. Haven't had the time to investigate why yet.

mgermain avatar Feb 17 '17 19:02 mgermain

Do you get an error? Or does it just not restart?

bartvm avatar Feb 17 '17 19:02 bartvm

Without the -V on Mammouth some part of the environment is not loaded.

mgermain avatar Feb 22 '17 22:02 mgermain

That's the entire point of the flag, no? But smart-dispatch shouldn't rely on the environment as set outside of the job. What environment variable does it try to access?

bartvm avatar Feb 22 '17 22:02 bartvm

No, the point of the flag is to not copy the current environment. Your system still has to load the environment of the user in the job. It does on Helios and it does it weirdly and partially on mp2.

It does not try to access anything, I just saw that some modules are missing when -V is used. I have to investigate why.

mgermain avatar Feb 23 '17 19:02 mgermain

That sounds weird... -V shouldn't affect at all how the job's environment is set up beyond not importing environment variables from the launch environment. It's possible that Mammouth doesn't load .bashrc at the beginning of a job. But in that case it's better to source .bashrc explicitly in the generated PBS file, instead of relying on exporting with -V IMO.

bartvm avatar Feb 23 '17 19:02 bartvm