FatesColdTwoStream tests rely on Python module in .bash_profile

[samsrabin]

Brief summary of bug

The Izumi FatesColdTwoStream and FatesColdTwoStreamNoCompFixedBioGeo tests from aux_clm fail, if called with run_sys_tests, unless Python is loaded in a user's .bash_profile (or presumably the respective profile files for other shells). The failure happens during the CREATE_NEWCASE step, during the call of shell_commands.

See this error reported by @slevis-lmwg during testing of during #2334, as well as subsequent messages from me.

General bug information

CTSM version you are using: ctsm5.1.dev164

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: n/a

Details of bug

The user can solve this by adding module load lang/python/3.7.0 to their .bash_profile, but it would be better to solve this permanently rather than relying on the user figuring this out.

The problem is that shell_commands in these tests calls the script $FATESDIR/tools/modify_fates_paramfile.py, which depends on numpy, scipy, etc. It would be more appropriate for that script to be called in the run_phase of a new SystemTest—call it MODIFYFATESPARAMF. This would make it so that, as long as the user calls run_sys_tests with ctsm_pylib loaded, everything necessary will be available. @glemieux, what do you think?

Important details of your setup / configuration so we can reproduce the bug

On Izumi, in bash, with no module load commands in my .bash_profile, I set up the failing test like so:

./run_sys_tests --skip-compare --skip-generate -t ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream

Important output or errors that show the problem

TestStatus.log:

RUN: /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz/shell_commands
FROM: /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz
  stat: 1

  errput: Traceback (most recent call last):
  File "/home/samrabin/ctsm/src/fates/tools/modify_fates_paramfile.py", line 35, in <module>
    from scipy.io import netcdf as nc
ImportError: No module named scipy.io
Leaving broken case dir /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz
ERROR: Command: '/scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz/shell_commands' failed with error 'Traceback (most recent call last):
  File "/home/samrabin/ctsm/src/fates/tools/modify_fates_paramfile.py", line 35, in <module>
    from scipy.io import netcdf as nc
ImportError: No module named scipy.io' from dir '/scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz'

 ---------------------------------------------------
2024-01-24 13:23:13: CREATE_NEWCASE FAILED for test 'ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream'.
Command: /bin/python3 /home/samrabin/ctsm/cime/CIME/scripts/create_newcase.py --case /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz --res f45_f45_mg37 --compset I2000Clm50FatesRs --test --machine izumi --compiler nag --project P93300641  --output-root /scratch/cluster/samrabin/tests_0124-132302iz  --workflow default --srcroot /home/samrabin/ctsm --user-mods-dir /home/samrabin/ctsm/cime_config/testdefs/testmods_dirs/clm/FatesColdTwoStream
Output: Compset longname is 2000_DATM%GSWP3v1_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV
Compset specification file is /home/samrabin/ctsm/cime_config/config_compsets.xml
Automatically adding SESP to compset
Leaving broken case dir /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz
ERROR: locked: cannot make child group in file /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz/env_case.xml

 ---------------------------------------------------
2024-01-24 13:23:15: CREATE_NEWCASE FAILED for test 'ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream'.
Command: /bin/python3 /home/samrabin/ctsm/cime/CIME/scripts/create_newcase.py --case /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz --res f45_f45_mg37 --compset I2000Clm50FatesRs --test --machine izumi --compiler nag --project P93300641  --output-root /scratch/cluster/samrabin/tests_0124-132302iz  --workflow default --srcroot /home/samrabin/ctsm --user-mods-dir /home/samrabin/ctsm/cime_config/testdefs/testmods_dirs/clm/FatesColdTwoStream
Output: Compset longname is 2000_DATM%GSWP3v1_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV
Compset specification file is /home/samrabin/ctsm/cime_config/config_compsets.xml
Automatically adding SESP to compset
Leaving broken case dir /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz
ERROR: locked: cannot make child group in file /scratch/cluster/samrabin/tests_0124-132302iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0124-132302iz/env_case.xml

 ---------------------------------------------------

[samsrabin]

See also discussion from CTSM SE meeting here.

[ekluzek]

Note, this starts showing up in ctsm5.3.003 and after for SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_intel.clm-FatesColdHydro because of a similar addition to shell_commands for it. It worked for @glemieux for ctsm5.3.003 because of his startup shell options.

[ekluzek]

This was resolved in ctsm5.3.014