Unable to compile cam6_3_154 with nvhpc/24.3 on Derecho

[sjsprecious]

What happened?

When I checked out the CAM tag cam6_3_154 with the ccs_config0.0.99 tag, I failed to build a F2000dev test on Derecho. The error message in the cesm.bldlog.* file looks like:

Building a single executable version of target coupled model
ERROR: Command gmake exec_se -j 16 EXEC_SE=../../cesm.exe COMP_NAME=driver CIME_MODEL=cesm  SMP=FALSE CASEROOT="/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154" CASETOOLS="/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools" CIMEROOT="/glade/derecho/scratch/sunjian/cam6_3_154/cime" SRCROOT="/glade/derecho/scratch/sunjian/cam6_3_154" COMP_INTERFACE="nuopc" COMPILER="nvhpc" DEBUG="FALSE" EXEROOT="/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld" RUNDIR="/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/run" INCROOT="/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include" LIBROOT="/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib" MACH="derecho" MPILIB="mpich" NINST_VALUE="c1a1l1i1o1r1" OS="CNL" PIO_VERSION=2 SHAREDLIBROOT="/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld" BUILD_THREADED="FALSE" USE_ESMF_LIB="TRUE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam_dev  -rad rrtmg  -pcols 16" COMP_ATM="cam" COMP_ICE="cice" COMP_GLC="sglc" COMP_LND="clm" COMP_OCN="docn" COMP_ROF="mosart" COMP_WAV="swav" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"  GLC_PRESENT=FALSE WAV_PRESENT=FALSE ESP_PRESENT=FALSE IAC_PRESENT=FALSE -f /glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile  failed rc=2
out=/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/mkSrcfiles
cp -f Filepath Deppath
/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/mkDepends  Deppath Srcfiles > Depends
ftn -c -I. -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/fox/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/dshr -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/zlnp/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/atm/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ice/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ocn/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/glc/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/rof/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/wav/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/esp/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj -I -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/clm/obj -I. -I/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/SourceMods/src.drv -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include -O -Mnofma -i4 -gopt  -time -Mextend -byteswapio -Mflushz -Kieee -target=zen3 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/3c7o/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/nvhpc/24.3/i6rj/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/nvhpc/24.3/dcds/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16  -DCPRPGI -DLINUX -DHAVE_GETTID -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DMED_PRESENT -DPIO2 -Mfree -DUSE_CONTIGUOUS= /glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator/med_kind_mod.F90
ftn -c -I. -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/fox/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/dshr -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/zlnp/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/atm/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ice/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ocn/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/glc/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/rof/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/wav/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/esp/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj -I -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/clm/obj -I. -I/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/SourceMods/src.drv -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include -O -Mnofma -i4 -gopt  -time -Mextend -byteswapio -Mflushz -Kieee -target=zen3 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/3c7o/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/nvhpc/24.3/i6rj/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/nvhpc/24.3/dcds/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16  -DCPRPGI -DLINUX -DHAVE_GETTID -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DMED_PRESENT -DPIO2 -Mfree -DUSE_CONTIGUOUS= /glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn/shr_flux_mod.F90
ftn -c -I. -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/fox/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/dshr -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/zlnp/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/atm/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ice/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ocn/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/glc/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/rof/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/wav/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/esp/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj -I -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/clm/obj -I. -I/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/SourceMods/src.drv -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include -O -Mnofma -i4 -gopt  -time -Mextend -byteswapio -Mflushz -Kieee -target=zen3 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/3c7o/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/nvhpc/24.3/i6rj/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/nvhpc/24.3/dcds/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16  -DCPRPGI -DLINUX -DHAVE_GETTID -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DMED_PRESENT -DPIO2 -Mfree -DUSE_CONTIGUOUS= /glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver/esm_time_mod.F90
ftn -c -I. -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/fox/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/dshr -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/zlnp/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/atm/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ice/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ocn/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/glc/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/rof/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/wav/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/esp/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj -I -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/clm/obj -I. -I/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/SourceMods/src.drv -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include -O -Mnofma -i4 -gopt  -time -Mextend -byteswapio -Mflushz -Kieee -target=zen3 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/3c7o/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/nvhpc/24.3/i6rj/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/nvhpc/24.3/dcds/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16  -DCPRPGI -DLINUX -DHAVE_GETTID -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DMED_PRESENT -DPIO2 -Mfree -DUSE_CONTIGUOUS= /glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver/t_driver_timers_mod.F90
ftn -c -I. -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/fox/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/CDEPS/dshr -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/c1a1l1i1o1r1/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/zlnp/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/atm/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ice/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/ocn/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/glc/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/rof/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/wav/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/esp/obj -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj -I -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/include -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/nuopc/esmf/clm/obj -I. -I/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/SourceMods/src.drv -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../mediator -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/lib/include -O -Mnofma -i4 -gopt  -time -Mextend -byteswapio -Mflushz -Kieee -target=zen3 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/nvhpc/24.3/2u4m/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/nvhpc/24.3/3c7o/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/nvhpc/24.3/i6rj/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/nvhpc/24.3/jqjr/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/nvhpc/24.3/dcds/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16  -DCPRPGI -DLINUX -DHAVE_GETTID -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DMED_PRESENT -DPIO2 -Mfree -DUSE_CONTIGUOUS= /glade/derecho/scratch/sunjian/cam6_3_154/components/cmeps/cime_config/../cesm/driver/util.F90
-I<incdir>          Add directory to include file search path
-I<incdir>          Add directory to include file search path
-I<incdir>          Add directory to include file search path
-I<incdir>          Add directory to include file search path
-I<incdir>          Add directory to include file search path
err=cat: Srcfiles: No such file or directory
Warning: -target=zen3 option ignored.
Warning: -target=zen3 option ignored.
Warning: -target=zen3 option ignored.
Warning: -target=zen3 option ignored.
Warning: -target=zen3 option ignored.
nvfortran-Error-Switch -I expects an argument
gmake: *** [/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile:1001: med_kind_mod.o] Error 1
gmake: *** Waiting for unfinished jobs....
nvfortran-Error-Switch -I expects an argument
gmake: *** [/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile:1001: shr_flux_mod.o] Error 1
nvfortran-Error-Switch -I expects an argument
gmake: *** [/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile:1001: t_driver_timers_mod.o] Error 1
nvfortran-Error-Switch -I expects an argument
nvfortran-Error-Switch -I expects an argument
gmake: *** [/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile:1001: esm_time_mod.o] Error 1
gmake: *** [/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/Tools/Makefile:1001: util.o] Error 1

This error happens because there is an option -I without a path appended to it somehow.

When I checked out the CAM tag cam6_3_153 with the ccs_config0.0.99 tag, I was able to build the same F2000dev case on Derecho successfully.

What are the steps to reproduce the bug?

./create_newcase --case /path_to_case --mach derecho --res ne30pg3_ne30pg3_mg17 --compset F2000dev --compiler nvhpc --ngpus-per-node 4 --gpu-type a100 --gpu-offload openacc cd /path_to_case ./case.setup ./case.build

What CAM tag were you using?

cam6_3_154

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

NVHPC

Path to a case directory, if applicable

/glade/derecho/scratch/sunjian/cam6/case/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154

Will you be addressing this bug yourself?

No

Extra info

The cesm error log for cam6_3_154 could be found on Derecho at /glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/cesm.bldlog.240503-115432.

The successful build of cam6_3_153 could be found on Derecho at /glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_153.

[cacraigucar]

@brian-eaton - cam6_3_154 is the gust updates tag. Since you are down that rabbit hole right now, I'm wondering if you might have seen this?

[brian-eaton]

I've taken a look at the cesm.bldlog files from both runs. The compile commandline for med_kind_mod.F90 in the run that fails (in the output posted above) contains

...
-I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/iac/obj
-I
-I/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154/bld/nvhpc/mpich/nodebug/nothreads/nuopc/include
...

while the run that succeeds does not have any -I between the .../iac/obj and .../nuopc/include directories. Looking at cime/CIME/Tools/Makefile at the code responsible for constructing the INCLDIR macro I see that between the code that sets those directories before and after the -I is the following:

ifdef PIO_LIBDIR
  ifeq ($(PIO_VERSION),$(PIO_VERSION_MAJOR))
    INCLDIR += -I$(PIO_INCDIR)
...

PIO_VERSION is being set in the gmake command, but I don't see either PIO_VERSION_MAJOR or PIO_INCDIR being set. That would result in the same INCLDIR as the successful run. The INCLDIR that includes -I could result if PIO_VERSION_MAJOR were defined to be 2 and PIO_INCDIR was not set. I don't know where PIO_VERSION_MAJOR is normally set, but is there any chance that it's somehow set in the script used for the failing run?

[sjsprecious]

Hi @brian-eaton , thank you very much for tracking down this issue and providing so many helpful details. On Derecho, once we load the parallelio/2.6.2 module, the PIO_VERSION_MAJOR environment variable is set to 2.

I also did not find where the PIO_INCDIR was set. Maybe @fischer-ncar or @jedwards4b knows it?

[fischer-ncar]

Hi @sjsprecious, I believe that PIO is in the ESMF library. So the PIO_INCDIR doesn't need to be set, just the ESMF include dir would need to be set.

[brian-eaton]

The successful run in cam6_3_153 didn't have PIO_INCDIR set either. Why is that run set up differently than the run with cam6_3_154?

[sjsprecious]

Thanks @fischer-ncar for your information. That is good to know.

By comparing the changes between cam6_3_153 and cam6_3_154, I see:

• cmeps is updated from tag 0.14.49 to 0.14.59.
• cdeps is updated from tag 1.0.26 to 1.0.31.

Looking at the changes for cdeps (https://github.com/ESCOMP/CDEPS/compare/97a2da6..df7ccea), I see the following new settings related to PIO:

• https://github.com/ESCOMP/CDEPS/blob/df7ccea66fd05cd59a822aff67088b7fef5b4169/dglc/CMakeLists.txt#L20.

Looking at the changes for cmeps (https://github.com/ESCOMP/CMEPS/compare/d2a08f0..d6dc571), I see the following new settings related to PIO:

• https://github.com/ESCOMP/CMEPS/blob/d6dc5710f7ec82315256364bdb6febef565ea507/cime_config/buildexe#L114-L116
• https://github.com/ESCOMP/CMEPS/blob/d6dc5710f7ec82315256364bdb6febef565ea507/mediator/med_phases_cdeps_mod.F90#L1-L291 (a newly added file where PIO is used)

Could one of them lead to the different behavior between cam6_3_153 and cam6_3_154 here? Thanks.

[fischer-ncar]

@sjsprecious, which cime tag are you using. I was able to get your test to work with the latest cesm2_3_alpha17e tag. That's using cime6.0.238_httpsbranch01.

[sjsprecious]

Hi @fischer-ncar , thanks for your quick reply. I used cime6.0.217_httpsbranch02 tag in cam6_3_153 and cam6_3_154.

I switched to cime6.0.238_httpsbranch01 in cam6_3_154 and still got the same error (/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.aer_bug_154_v2/bld/cesm.bldlog.240508-145200).

However, I checked out the cesm2_3_alpha17e tag on Derecho and I could confirm that it was built successfully with the same compset setup (/glade/derecho/scratch/sunjian/cam6_run/F2000dev.ne30pg3_ne30pg3_mg17.derecho.nvhpc.gpu04_pcols00016_mpi0128_thread001_rrtmg.cesm/bld).

Thus it might be caused by something other than cime?

[sjsprecious]

Hi @brian-eaton and @fischer-ncar , it turns out that by switching to the latest ccs_config_cesm0.0.106 tag, I was able to build the same case with the cam6_3_154 tag on Derecho.

The latest ccs_config tag includes a bug fix for PIO and NVHPC compiler specifically on Derecho (https://github.com/ESMCI/ccs_config_cesm/pull/168), thanks to @jedwards4b .

However, it is still a mystery to me why cam6_3_153 succeeds but cam6_3_154 fails.

[sjsprecious]

@brian-eaton I think I find the reason about the failure of cam6_3_154:

• In cam6_3_154, the PIO_LIBDIR is explicitly defined in CMEPS (https://github.com/ESCOMP/CMEPS/blob/d6dc5710f7ec82315256364bdb6febef565ea507/cime_config/buildexe#L114-L116). Therefore, the section you pointed out at cime/CIME/Tools/Makefile (https://github.com/ESMCI/cime/blob/master/CIME/Tools/Makefile#L455-L463) will be evaluated and it leads to adding an empty path since PIO_INCDIR is not defined for the NVHPC compiler in ccs_config_cesm0.0.99, which is later fixed in ccs_config_cesm0.0.106.
• In cam6_3_153, the PIO_LIBDIR is not explicitly defined. Thus the same section in cime/CIME/Tools/Makefile is not evaluated at all and no empty path will be appended to the INCLDIR. This is why the compilation could succeed.

If this makes sense to you, I think the conclusion here is to use the latest ccs_config_cesm0.0.106 tag instead and I am fine to close this issue. Thanks for your time and help.

[brian-eaton]

@sjsprecious, Nice work tracking this down. Cheryl will soon be updating CAM's externals and the ccs_config_cesm0.0.106 tag will come in with that update.

[sjsprecious]

Thanks @brian-eaton . That sounds great!