CTSM
CTSM copied to clipboard
ESCOMP
FUNITCTSM tests fail on Izumi due to a ccs_config update
Brief summary of bug
With ESCOMP/CTSM#3125, the FUNITCTSM tests fail on Izumi due to an update for ccs_config in ccs_config_cesm1.0.42
General bug information
CTSM version you are using: ctsm5.3.045-181-g54e640b6
Does this bug cause significantly incorrect results in the model's science? [Yes / No]
Configurations affected: Just the PF unit testing and only on Izumi
Details of bug
FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel
fails at build time (as well as when run by hand under the src directory).
This starts with ccs_config_cesm1.0.42
Important output or errors that show the problem
The error on build looks like this:
... Done. Results in test_filterBuilder.F90
... Done. Results in test_unittestArray.F90
... Done. Results in test_quadratic.F90
... Done. Results in test_unittestSubgrid.F90
... Done. Results in test_annual_flux_dribbler.F90
... Done. Results in test_find_k_max_indices.F90
... Done. Results in test_truncate_small_values.F90
... Done. Results in test_clm_time_manager.F90
[ 45%] Generating test_convert_to_logical.F90
[ 49%] Building Fortran object clm_utils_test/annual_flux_dribbler_test/CMakeFiles/annual_flux_dribbler.dir/annual_flux_dribbler_driver.F90.o
[ 50%] Building Fortran object clm_unit_test_shr_test/unittestSubgrid_test/CMakeFiles/unittestSubgrid.dir/test_unittestSubgrid.F90.o
[ 50%] Building Fortran object clm_unit_test_shr_test/unittestFilterBuilder_test/CMakeFiles/unittestFilterBuilder.dir/test_filterBuilder.F90.o
[ 50%] Building Fortran object clm_utils_test/clm_time_manager_test/CMakeFiles/clm_time_manager.dir/clm_time_manager_driver.F90.o
[ 50%] Building Fortran object clm_utils_test/quadratic_test/CMakeFiles/quadratic.dir/quadratic_driver.F90.o
[ 50%] Building Fortran object clm_unit_test_shr_test/unittestArray_test/CMakeFiles/unittestArray.dir/test_unittestArray.F90.o
[ 50%] Building Fortran object clm_utils_test/numerics_test/CMakeFiles/numerics.dir/numerics_driver.F90.o
Processing file ../../../../../../../../../fs/cgd/data0/erik/ctsm_worktree/externals_update/src/utils/test/array_utils_test/test_convert_to_logical.pf
... Done. Results in test_convert_to_logical.F90
[ 50%] Building Fortran object clm_utils_test/array_utils_test/CMakeFiles/array_utils.dir/array_utils_driver.F90.o
[ 50%] Building Fortran object clm_utils_test/quadratic_test/CMakeFiles/quadratic.dir/test_quadratic.F90.o
[ 50%] Building Fortran object clm_utils_test/annual_flux_dribbler_test/CMakeFiles/annual_flux_dribbler.dir/test_annual_flux_dribbler.F90.o
[ 50%] Building Fortran object clm_utils_test/numerics_test/CMakeFiles/numerics.dir/test_truncate_small_values.F90.o
[ 51%] Building Fortran object clm_utils_test/clm_time_manager_test/CMakeFiles/clm_time_manager.dir/test_clm_time_manager.F90.o
[ 51%] Building Fortran object clm_utils_test/array_utils_test/CMakeFiles/array_utils.dir/test_convert_to_logical.F90.o
/scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.20250528_122148_6b5ugm/bld/__command_line_test__/__command_line_test__/clm_utils_test/annual_flux_dribbler_test//fs/cgd/data0/erik/ctsm_worktree/externals_update/src/utils/test/annual_flux_dribbler_test/test_annual_flux_dribbler.pf(133): warning #5462: Global name too long, shortened from: test_annual_flux_dribblerdeltaaddedmidyear_ignoredinlatertimesteps_mp_DELTA_SHOULD_BE_IGNORED to: st_annual_flux_dribblerdeltaaddedmidyear_ignoredinlatertimesteps_mp_DELTA_SHOULD_BE_IGNORED
real(r8), parameter :: delta_should_be_ignored(1) = [3.e6_r8]
---------------------------^
[ 51%] Building Fortran object clm_unit_test_shr_test/unittestFilterBuilder_test/CMakeFiles/unittestFilterBuilder.dir/unittestFilterBuilder_driver.F90.o
[ 51%] Linking Fortran executable unittestFilterBuilder
ifort: error #10236: File not found: '$(NETCDF_PATH)/lib/libnetcdff.a'
make[2]: *** [clm_unit_test_shr_test/unittestFilterBuilder_test/CMakeFiles/unittestFilterBuilder.dir/build.make:121: clm_unit_test_shr_test/unittestFilterBuilder_test/unittestFilterBuilder] Error 1
make[1]: *** [CMakeFiles/Makefile2:1898: clm_unit_test_shr_test/unittestFilterBuilder_test/CMakeFiles/unittestFilterBuilder.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
A clarification... From what I can tell, I get the same error in .../run/funit.log using
git describe: alpha-ctsm5.4.CMIP7.01.ctsm5.3.052-6-g3a8c43262
BUT ./cs.status shows the test failing in the RUN phase.
Also for me in #3501, passing in MODEL_BUILD, still failing in RUN git describe: alpha-ctsm5.4.CMIP7.11.ctsm5.3.075-10-g1283fb7bd so I will update ExpectedTestFails.xml accordingly.
I initially thought that #3577 solved this as I could run the PFUNIT testing in a sandbox on izumi, with the change. However, when running in tests it fails, and it fails when I delete the unit_tests.temp build directory. So this means if I have an existing build from when it was working -- it'll still work now.
What I get as an error now is the following (in either testing under src or in the FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel unit test) is:
2025-10-31 12:50:42: ERROR: RUN FAIL for FUNIT
---------------------------------------------------
2025-10-31 12:52:27: Compiler is intel
copying /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/ccs_config/machines/cmake_macros/../izumi/nag_izumi.cmake to /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp. izumi_intel.GC.ctsm5384izlist/bld/cmake_macros
copying /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/ccs_config/machines/cmake_macros/../izumi/pgi_izumi.cmake to /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp. izumi_intel.GC.ctsm5384izlist/bld/cmake_macros
copying /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/ccs_config/machines/cmake_macros/../izumi/gnu_izumi.cmake to /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp. izumi_intel.GC.ctsm5384izlist/bld/cmake_macros
copying /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/ccs_config/machines/cmake_macros/../izumi/izumi.cmake to /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izum i_intel.GC.ctsm5384izlist/bld/cmake_macros
copying /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/ccs_config/machines/cmake_macros/../izumi/intel_izumi.cmake to /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50S p.izumi_intel.GC.ctsm5384izlist/bld/cmake_macros
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using PFUNIT_PATH: /fs/cgd/csm/tools/pFUnit/pFUnit4.7.0_izumi_Intel20.0.1_noMPI_noOpenMP
Setting NETCDF environment variable: /usr/local/netcdf-c-4.7.4-f-4.5.2-intel-cluster-20.0.1
==================================================
Running cmake for __command_line_test__/__command_line_test__.
==================================================
ERROR: Command: 'cmake -C Macros.cmake /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/src -DCIMEROOT=/fs/cgd/data0/erik/ctsm_worktree/b4b-dev/cime/scripts/fortran_unit_testing/../ .. -DSRC_ROOT=/fs/cgd/data0/erik/ctsm_worktree/b4b-dev -DCIME_CMAKE_MODULE_DIRECTORY=/fs/cgd/data0/erik/ctsm_worktree/b4b-dev/cime/CIME/non_py/src/CMake -DCMAKE_BUILD_TYPE=C ESM_DEBUG -DCMAKE_PREFIX_PATH=/fs/cgd/csm/tools/pFUnit/pFUnit4.7.0_izumi_Intel20.0.1_noMPI_noOpenMP -DUSE_MPI_SERIAL=ON -DENABLE_GENF90=ON -DCMAKE_PROGRAM_PATH=/fs/cgd/data0 /erik/ctsm_worktree/b4b-dev/cime/scripts/fortran_unit_testing/../../CIME/non_py/externals/genf90 -DOS=LINUX -DMACH=izumi -DCOMPILER=intel -DDEBUG=TRUE -DMPILIB=mpi-serial -D compile_threaded=FALSE -DCASEROOT=/scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.ctsm5384izlist/bld' failed with error 'loading initial cache file Macros.cmake
No macro file found: /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.ctsm5384izlist/bld/cmake_macros/LINUX.cmake
No macro file found: /scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.ctsm5384izlist/bld/cmake_macros/intel_LINUX.cmake
CMake Error at /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/cime/CIME/non_py/src/CMake/CIME_initial_setup.cmake:6 (include):
include could not find requested file:
/fs/cgd/data0/erik/ctsm_worktree/b4b-dev/src/Macros.cmake
Call Stack (most recent call first):
CMakeLists.txt:4 (include)
CMake Error at /fs/cgd/data0/erik/ctsm_worktree/b4b-dev/cime/CIME/non_py/src/CMake/CIME_initial_setup.cmake:8 (message):
You must generate a Macros.cmake file using CIME's configure
Call Stack (most recent call first):
CMakeLists.txt:4 (include)
-- Configuring incomplete, errors occurred!' from dir '/scratch/cluster/erik/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.ctsm5384izlist/bld/__command_line_test__ /__command_line_test__'
So this continues to be a problem.