ESCOMP
Python testing was failing for me on Derecho
Brief summary of bug
I was having problems running the python unit and system tests on Derecho. I saw it first in the latest version that I was bringing in, but then saw it for existing tags that I had already shown to work.
So I'm putting it here for visibility in case someone else runs into this.
General bug information
CTSM version you are using: ctsm5.3.041
Does this bug cause significantly incorrect results in the model's science? ?
Configurations affected: Python testing and also tools?
Details of bug
Running the python testing started failing for me in the creation of ctsm5.3.051, and I quickly saw the same trouble in previous tags. And this was in cases I knew had worked previously.
I had @slevis-lmwg show that it worked for him. And I compared my module environment and the ctsm_pylib environment between the two of us. The differences were small and it wasn't obvious why it wasn't working for me vs. him.
Important details of your setup / configuration so we can reproduce the bug
What I did to finally get it working:
I did this more than once, but I had to completely delete the complete ctsm_pylib environment and then reinstall it again with ./py_env_create. Just updating the current environment wasn't enough. And I did the removal yesterday to no avail, but doing the same thing today, it magically worked. I also did an explicit removal with conda commands, but presumably using ./py_env_create options to remove the previous environment would have worked. But, I can't say because I didn't do it that way. Another difference between the days is that yesterday I was testing this in ctsm5.3.041 while today, I did it in ctsm5.3.052 (there is the addition of a pip requirement in ctsm5.3.052 that maybe made a difference?).
Important output or errors that show the problem
Here's the kind of messaging I was getting.
./run_ctsm_py_tests --unit
...............EEEEEEEEEEEEEEEEEEEEE.......................................................................................................................E...E..................................................................................................................................EEEEEE
======================================================================
ERROR: test_dompft_and_splitcropland_fails (test.test_unit_fsurdat_modifier.TestFSurdatModifier.test_dompft_and_splitcropland_fails)
test that setting dompft crop with evenly_split_cropland True fails gracefully
----------------------------------------------------------------------
Traceback (most recent call last):
File "/glade/work/erik/ctsm_worktrees/external_updates/python/ctsm/test/test_unit_fsurdat_modifier.py", line 65, in setUp
my_data = xr.open_dataset(self._fsurdat_in)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/api.py", line 667, in open_dataset
engine = plugins.guess_engine(filename_or_obj)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/plugins.py", line 194, in guess_engine
raise ValueError(error_msg)
ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html
======================================================================
ERROR: test_optional_only_true_and_false (test.test_unit_fsurdat_modifier.TestFSurdatModifier.test_optional_only_true_and_false)
test that optional settings can only be true or false
----------------------------------------------------------------------
Traceback (most recent call last):
File "/glade/work/erik/ctsm_worktrees/external_updates/python/ctsm/test/test_unit_fsurdat_modifier.py", line 65, in setUp
my_data = xr.open_dataset(self._fsurdat_in)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/api.py", line 667, in open_dataset
engine = plugins.guess_engine(filename_or_obj)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/plugins.py", line 194, in guess_engine
raise ValueError(error_msg)
ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html
======================================================================
ERROR: test_read_subgrid (test.test_unit_fsurdat_modifier.TestFSurdatModifier.test_read_subgrid)
test a simple read of subgrid
----------------------------------------------------------------------
Traceback (most recent call last):
File "/glade/work/erik/ctsm_worktrees/external_updates/python/ctsm/test/test_unit_fsurdat_modifier.py", line 65, in setUp
my_data = xr.open_dataset(self._fsurdat_in)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/api.py", line 667, in open_dataset
engine = plugins.guess_engine(filename_or_obj)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/plugins.py", line 194, in guess_engine
raise ValueError(error_msg)
ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html
======================================================================
ERROR: test_read_subgrid_allglacier (test.test_unit_fsurdat_modifier.TestFSurdatModifier.test_read_subgrid_allglacier)
test a read of subgrid that's for all glacier
----------------------------------------------------------------------
Traceback (most recent call last):
.
.
.
======================================================================
ERROR: test_importcoord2d_rename_dim (test.test_unit_utils_import_coord.TestUtilsImportCoord.test_importcoord2d_rename_dim)
Tests renaming of an imported 2-d lat/lon variable
----------------------------------------------------------------------
Traceback (most recent call last):
File "/glade/work/erik/ctsm_worktrees/external_updates/python/ctsm/test/test_unit_utils_import_coord.py", line 121, in test_importcoord2d_rename_dim
ds = xr.open_dataset(self._2d_lonlat_file)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/api.py", line 667, in open_dataset
engine = plugins.guess_engine(filename_or_obj)
File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.13/site-packages/xarray/backends/plugins.py", line 194, in guess_engine
raise ValueError(error_msg)
ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html
----------------------------------------------------------------------
Ran 296 tests in 6.762s
FAILED (errors=29)
