GEOS-Chem 14.4 and dependencies

[lizziel]

This PR updates GEOS-Chem from version 14.1.2 to 14.4.3 It also includes updating HEMCO_CESM to 2.0.0 and HEMCO to 3.9.0. In addition, there are two new submodules in CAM: (1) Cloud-J 7.7.3, now used in GEOS-Chem for computing J-values, and (2) HETP v1.1, an improved ISORROPIA for aerosol thermodynamics in GEOS-Chem.

This PR supercedes https://github.com/ESCOMP/CAM/pull/996 which was updating to an older version of GEOS-Chem.

Closes https://github.com/ESCOMP/CAM/issues/959

[lkemmons]

ACOM (e.g., @tilmes, Doug Kinnison) think Cloud-J would be a great option to include in CESM3, as an alternative to TUV-x for now.

[lizziel]

The tests failed due to git-fleximod errors. I am looking into them now.

s                hemco hemco-cesm2_0_hemco_3_9_0 7bd8358 is out of sync with .gitmodules hemco-cesm2_0_hemco3_9_0
          e                HEMCO has no fxtag defined in .gitmodules, module at 3.9.0
                    e geos-chem-shared-docs has no fxtag defined in .gitmodules, module at cbfb5cb
            rte-rrtmgp at tag v1.7
           rrtmgp-data at tag v1.8
                  mpas at hash b8c33daa4
                 cosp2 at tag v2.1.4cesm
                 clubb at tag clubb_4ncar_20231115_540[63](https://github.com/ESCOMP/CAM/actions/runs/10201686947/job/28224132055?pr=1093#step:3:64)50
                     source_cism at tag cism_main_2.01.013
          e    CIME/non_py/cprnc has no fxtag defined in .gitmodules, module at v1.0.6

                         fox at tag 4.1.2.1
                      genf90 at hash 48169[65](https://github.com/ESCOMP/CAM/actions/runs/10201686947/job/28224132055?pr=1093#step:3:66)
                         src at tag cice6_5_0_20231221
                e              icepack has no fxtag defined in .gitmodules, module at f6ff8f7
                       fates at tag sci.1.73.0_api.35.0.0
                         fms at tag dev/ncar_0.0.1

[lizziel]

This issue is .gitmodules for cice, cime, and hemco_config is configured for regular git submodules without the additional parameters required by fleximod. I am using the cime and cice versions that are in cam_development, and the new hemco_cesm tag is built on the tag used in cam_development too. Let me know if you need me to do anything.

[fvitt]

Suggested changes to test/system/TR8.sh

diff --git a/test/system/TR8.sh b/test/system/TR8.sh
index 498c9bb5..cbdb4004 100755
--- a/test/system/TR8.sh
+++ b/test/system/TR8.sh
@@ -58,12 +58,12 @@ fi
 #Check Chemistry
 if [ -d "${CAM_ROOT}/components/cam" ]; then
 
-ruby $ADDREALKIND_EXE -r r8 -l 1 -d $CAM_ROOT/components/cam/src/chemistry -s geoschem
+ruby $ADDREALKIND_EXE -r r8 -l 1 -d $CAM_ROOT/components/cam/src/chemistry -s geoschem/geoschem_src,cloud_j,hetp
 rc=`expr $? + $rc`
 
 else
 
-ruby $ADDREALKIND_EXE -r r8 -l 1 -d $CAM_ROOT/src/chemistry -s geoschem
+ruby $ADDREALKIND_EXE -r r8 -l 1 -d $CAM_ROOT/src/chemistry -s geoschem/geoschem_src,cloud_j,hetp
 rc=`expr $? + $rc`
 
 fi

[lizziel]

When I run submit aux_cam test suite I get a lot of errors because I am on derecho instead of izumi. Is it still useful for me to test with it? Update: it is running derecho tests instead

[fvitt]

When I run submit aux_cam test suite I get a lot of errors because I am on derecho instead of izumi. Is it still useful for me to test with it? Update: it is running derecho tests instead

test_driver.sh in test/system/ will launch the aux_cam tests on derecho and izumi.

This is how I submit the tests on derecho:

env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_018_intel test_driver.sh -f

[lizziel]

I now have lots of tests in ~elundgren/scratch. Could you tell me what tests to look at? Or do you run a script to evaluate all of them at once?

[fvitt]

I now have lots of tests in ~elundgren/scratch. Could you tell me what tests to look at? Or do you run a script to evaluate all of them at once?

To get a summary of the test results try thist:

cs.status.20240809_111703_3ffrar -s

I see that you have a izumi tests as well as derecho tests. You must be launching the tests differntly.

When I use the test_driver.sh script it creates an aux_cam sub dir under my scratch. For an example, see

/glade/derecho/scratch/fvitt/aux_cam_20240807162715

[lizziel]

@fvitt, here are the aux_cam test results. Let me know if we should be concerned about any of these. I am not sure why some are still pending given this was done last week.

  PASS ERC_D_Ln9.f19_f19_mg17.QPC6.derecho_intel.cam-outfrq3s_cosp
  PASS ERC_D_Ln9.f19_f19_mg17.QPMOZ.derecho_intel.cam-outfrq3s
  PASS ERC_D_Ln9.f19_f19_mg17.QPX2000.derecho_intel.cam-outfrq3s
  PASS ERC_D_Ln9.ne16_ne16_mg17.FADIAB.derecho_intel.cam-terminator
  PASS ERC_D_Ln9.ne16_ne16_mg17.QPC5HIST.derecho_intel.cam-outfrq3s_usecase
  PASS ERC_D_Ln9_P144x1.ne16pg3_ne16pg3_mg17.QPC6HIST.derecho_intel.cam-outfrq3s_ttrac_usecase
  PASS ERC_D_Ln9.T42_T42_mg17.FDABIP04.derecho_intel.cam-outfrq3s_usecase
  PASS ERC_D_Ln9.T42_T42_mg17.FHS94.derecho_intel.cam-outfrq3s_usecase
  PASS ERI_D_Ln18.f45_f45_mg37.QPC41850.derecho_intel.cam-co2rmp_usecase
  PASS ERP_D_Ln9.f19_f19_mg17.QPC6.derecho_intel.cam-outfrq9s
  PASS ERP_D_Ln9.ne30pg3_ne30pg3_mg17.FLTHIST.derecho_intel.cam-outfrq9s
  PASS ERP_D_Ln9.ne30pg3_ne30pg3_mg17.FLTHIST.derecho_intel.cam-outfrq9s_rrtmgp
  PASS ERP_D_Ln9_P64x2.f09_f09_mg17.QSC6.derecho_intel.cam-outfrq9s
  PEND ERP_Ld3.f09_f09_mg17.FWHIST.derecho_intel.cam-reduced_hist1d
  PEND ERP_Lh12.f19_f19_mg17.FW4madSD.derecho_intel.cam-outfrq3h
  PASS ERP_Ln9.C96_C96_mg17.F2000climo.derecho_intel.cam-outfrq9s_mg3
  PASS ERP_Ln9.f09_f09_mg17.F1850.derecho_intel.cam-outfrq9s
  PASS ERP_Ln9.f09_f09_mg17.F2000climo.derecho_intel.cam-outfrq9s
  PASS ERP_Ln9.f09_f09_mg17.F2010climo.derecho_intel.cam-outfrq9s
  PEND ERP_Ln9.f09_f09_mg17.FCSD_HCO.derecho_intel.cam-outfrq9s
  PASS ERP_Ln9.f09_f09_mg17.FHIST_BDRD.derecho_intel.cam-outfrq9s
  PEND ERP_Ln9.f19_f19_mg17.FWsc1850.derecho_intel.cam-outfrq9s
  PEND ERP_Ln9.ne30pg3_ne30pg3_mg17.FCnudged.derecho_intel.cam-outfrq9s
  PEND ERP_Ln9.ne30pg3_ne30pg3_mg17.FW2000climo.derecho_intel.cam-outfrq9s
  PASS ERP_Ln9_P24x3.f45_f45_mg37.QPWmaC6.derecho_intel.cam-outfrq9s_mee_fluxes
  PEND ERS_Ld3.f10_f10_mg37.F1850.derecho_intel.cam-outfrq1d_14dec_ghg_cam7
  PEND ERS_Ln9.f09_f09_mg17.FX2000.derecho_intel.cam-outfrq9s
  PASS ERS_Ln9.f19_f19_mg17.FSPCAMS.derecho_intel.cam-outfrq9s
  PEND ERS_Ln9.f19_f19_mg17.FXSD.derecho_intel.cam-outfrq9s
  PASS ERS_Ln9.ne0TESTONLYne5x4_ne0TESTONLYne5x4_mg37.FADIAB.derecho_intel.cam-outfrq3s_refined
  PASS ERS_Ln9_P288x1.mpasa120_mpasa120.F2000climo.derecho_intel.cam-outfrq9s_mpasa120
  PASS ERS_Ln9_P36x1.mpasa480_mpasa480.F2000climo.derecho_intel.cam-outfrq9s_mpasa480
  PASS SCT_D_Ln7.T42_T42_mg17.QPC5.derecho_intel.cam-scm_prep
  PASS SMS_D_Ld2.f19_f19_mg17.QPC5HIST.derecho_intel.cam-volc_usecase
  PASS SMS_D_Ld5.f19_f19_mg17.PC4.derecho_intel.cam-cam4_port5d
  PASS SMS_D_Ln9.f09_f09_mg17.FCts2nudged.derecho_intel.cam-outfrq9s_leapday
  PASS SMS_D_Ln9.f09_f09_mg17.FCvbsxHIST.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.f09_f09_mg17.FSD.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4
  PASS SMS_D_Ln9.f19_f19_mg17.FXHIST.derecho_intel.cam-outfrq9s_amie
  PASS SMS_D_Ln9.f19_f19_mg17.QPC2000climo.derecho_intel.cam-outfrq3s_usecase
  PASS SMS_D_Ln9.f19_f19_mg17.QPC5M7.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.ne16_ne16_mg17.QPX2000.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.ne16pg3_ne16pg3_mg17.FX2000.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.ne30pg3_ne30pg3_mg17.FCts4MTHIST.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9.ne30pg3_ne30pg3_mg17.FMTHIST.derecho_intel.cam-outfrq9s
  FAIL SMS_D_Ln9_P1280x1.ne0ARCTICne30x4_ne0ARCTICne30x4_mt12.FHIST.derecho_intel.cam-outfrq9s
  FAIL SMS_D_Ln9_P1280x1.ne0CONUSne30x8_ne0CONUSne30x8_mt12.FCHIST.derecho_intel.cam-outfrq9s
  PASS SMS_D_Ln9_P1280x1.ne30pg3_ne30pg3_mg17.FCLTHIST.derecho_intel.cam-outfrq9s
  FAIL SMS_D_Ln9.T42_T42.FSCAM.derecho_intel.cam-outfrq9s
  PASS SMS_Ld1.f09_f09_mg17.FCHIST_GC.derecho_intel.cam-outfrq1d
  PEND SMS_Ld1.f09_f09_mg17.FW2000climo.derecho_intel.cam-outfrq1d
  PEND SMS_Ld1.ne30pg3_ne30pg3_mg17.FC2010climo.derecho_intel.cam-outfrq1d
  PASS SMS_Ld5.f09_f09_mg17.PC6.derecho_intel.cam-cam6_port_f09
  PEND SMS_Lh12.f09_f09_mg17.FCSD_HCO.derecho_intel.cam-outfrq3h
  PASS SMS_Lm13.f10_f10_mg37.F2000climo.derecho_intel.cam-outfrq1m
  PASS SMS_Ln9.f09_f09_mg17.F2010climo.derecho_intel.cam-nudging
  PEND SMS_Ln9.f09_f09_mg17.FW1850.derecho_intel.cam-reduced_hist3s
  PASS SMS_Ln9.f19_f19.F2000climo.derecho_intel.cam-silhs
  PASS SMS_Ln9.f19_f19_mg17.FHIST.derecho_intel.cam-outfrq9s_nochem
  PEND SMS_Ln9.ne30pg3_ne30pg3_mg17.FW2000climo.derecho_intel.cam-outfrq9s_rrtmgp

[lizziel]

When I run submit aux_cam test suite I get a lot of errors because I am on derecho instead of izumi. Is it still useful for me to test with it? Update: it is running derecho tests instead

test_driver.sh in test/system/ will launch the aux_cam tests on derecho and izumi.

This is how I submit the tests on derecho:

env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_018_intel test_driver.sh -f

Good to know. I submitted via this since it is what I use for geoschem category tests: qcmd -A UHAR0022 ./create_test --xml-category aux_cam -p UHAR0022

[cacraigucar]

When I run submit aux_cam test suite I get a lot of errors because I am on derecho instead of izumi. Is it still useful for me to test with it? Update: it is running derecho tests instead

test_driver.sh in test/system/ will launch the aux_cam tests on derecho and izumi. This is how I submit the tests on derecho:

`env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_018_intel test_driver.sh -f`

Good to know. I submitted via this since it is what I use for geoschem category tests: qcmd -A UHAR0022 ./create_test --xml-category aux_cam -p UHAR0022

Using the example that Francis provided, you can add a flag --cesm geoschem and that will run just the geoschem runs.

In summary: env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_018_intel test_driver.sh -f will run aux_cam jobs

env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_018_intel test_driver.sh -f --cesm geoschem will run geoschem jobs

The test_driver.sh script is how all AMP SEs run our regression testing.

[lizziel]

Thanks for the tips @cacraigucar. @fvitt, could you give me an update on where this PR stands. I see there are now conflicts due to new commits in cam_development. I can rebase and test again next week. Otherwise there are no remaining updates to add since I believe I addressed all of your comments. I am not sure about the test results above for non-GEOS-Chem runs and will need help deciphering those.

[lizziel]

@fvitt, I have rebased locally and will run tests on derecho and izumi prior to pushing. However, when I try to use test_driver.sh I get this error: ERROR: Must set the environment variable CAM_ACCOUNT

Should I set this to the GEOS-Chem project or to some other account?

[fvitt]

@fvitt, I have rebased locally and will run tests on derecho and izumi prior to pushing. However, when I try to use test_driver.sh I get this error: ERROR: Must set the environment variable CAM_ACCOUNT

Should I set this to the GEOS-Chem project or to some other account?

Yes, use the GEOS-Chem project number.

[lizziel]

@fvitt, now I am getting ERROR: Must specify CAM_FC. Do you have a sample environment file with env settings I should use with test_driver.sh? These are the ones I need based on the file:

  echo "Supported ENVIRONMENT variables"
  echo "BL_TESTDIR:        Default = none (used to set baseline compare dir)"
  echo "CAM_ACCOUNT:       Default = none"
  echo "CAM_BATCHQ:        Default = machine dependent"
  echo "CAM_FC:            Default = machine dependent"
  echo "CAM_INPUT_TESTS:   Default = tests_pretag_<machine>[_<compiler>]"
  echo "CAM_RESTART_TASKS: Default = 64"
  echo "CAM_RETAIN_FILES:  Default = FALSE"
  echo "CAM_ROOT:          Default = set relative to CAM_SCRIPTDIR"
  echo "NB: If script is not called as ./`basename ${0}`, CAM_ROOT must be specified"
  echo "CAM_SCRIPTDIR:     Default = <current directory>"
  echo "CAM_TAG:           Default = none (used to set CESM baseline dir)"
  echo "CAM_TASKS:         Default = (depends on system)"
  echo "CAM_TESTDIR:       Default = <user_scratch_dir>/test-driver.<jobid>"
  echo ""
  echo "Less common ENVIRONMENT variables"
  echo "CALDERA_BATCHQ:    Default = caldera"
  echo "CAM_RBOPTIONS:     Default = build_only"
  echo "CAM_SOFF:          Default = none (stop of first test fail if TRUE)"
  echo "CIME_MODEL:        Default = none (should be set to cesm)"
  echo "EMAIL:             Default = [email protected]"

[fvitt]

@lizziel Try submitting the tests on izumi like this:

env CAM_FC=NAG BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag test_driver.sh -f
env CAM_FC=GNU BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_gnu test_driver.sh -f

[lizziel]

Still not working...

$ env CAM_FC=NAG BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag ./test_driver.sh -f --cesm geoschem

CESM_BASELINE /fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag not found.  Check BL_TESTDIR for correct tag name.

[fvitt]

Still not working...

$ env CAM_FC=NAG BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag ./test_driver.sh -f --cesm geoschem

CESM_BASELINE /fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag not found.  Check BL_TESTDIR for correct tag name.

Did you try this on izumi?

[lizziel]

Ah, I did not notice you switched to izumi. I am trying to run tests using test_driver.sh on derecho prior to pushing. I do not have the code on izumi right now.

[lizziel]

Tests are now running on izumi. Most runs are still pending. However, I am already getting failures for all MEMCOMP NAG tests due to missing file cpl-mem.log:

[elundgren@izumi aux_cam_nag_20240906085937]$ ./cs.status.aux_cam_nag_20240906085937 | grep FAIL
    FAIL ERC_D_Ln9.f10_f10_mg37.QSPCAMS.izumi_nag.cam-outfrq3s MEMCOMP [Errno 2] No such file or directory: '/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag/ERC_D_Ln9.f10_f10_mg37.QSPCAMS.izumi_nag.cam-outfrq3s/cpl-mem.log'
    FAIL ERC_D_Ln9.T5_T5_mg37.QPC4.izumi_nag.cam-outfrq3s_usecase MEMCOMP [Errno 2] No such file or directory: '/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag/ERC_D_Ln9.T5_T5_mg37.QPC4.izumi_nag.cam-outfrq3s_usecase/cpl-mem.log'
    FAIL ERP_Ln9.ne5pg3_ne5pg3_mg37.QPC6.izumi_nag.cam-outfrq9s_clubbmf MEMCOMP [Errno 2] No such file or directory: '/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag/ERP_Ln9.ne5pg3_ne5pg3_mg37.QPC6.izumi_nag.cam-outfrq9s_clubbmf/cpl-mem.log'
    FAIL SMS_D_Ln7.T42_T42_mg17.QPSCAMC5.izumi_nag.cam-scmarm MEMCOMP [Errno 2] No such file or directory: '/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag/SMS_D_Ln7.T42_T42_mg17.QPSCAMC5.izumi_nag.cam-scmarm/cpl-mem.log'
    FAIL TMC_D.T5_T5_mg37.QPC5.izumi_nag.cam-ghgrmp_e8 MEMCOMP [Errno 2] No such file or directory: '/fs/cgd/csm/models/atm/cam/pretag_bl/cam6_4_027_nag/TMC_D.T5_T5_mg37.QPC5.izumi_nag.cam-ghgrmp_e8/cpl-mem.log'

[lizziel]

Izumi tests results, skipping lines containing MEMCOMP, TPUTCOMP, and PASS are as follows.

GNU tests all pass

[elundgren@izumi aux_cam_gnu_20240906095613]$ ./cs.status.aux_cam_gnu_20240906095613 | grep -v PASS | grep -v MEMCOMP | grep -v TPUTCOMP
aux_cam_gnu_20240906095613: 26 tests

NAG test fails one test which I am looking into

[elundgren@izumi aux_cam_nag_20240906085937]$ ./cs.status.aux_cam_nag_20240906085937 | grep -v PASS | grep -v MEMCOMP | grep -v TPUTCOMP
aux_cam_nag_20240906085937: 39 tests
  DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae (Overall: FAIL) details:
    FAIL DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae RUN time=10
    PEND DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae COMPARE_base_da

[cacraigucar]

Izumi tests results, skipping lines containing MEMCOMP, TPUTCOMP, and PASS are as follows.

GNU tests all pass

[elundgren@izumi aux_cam_gnu_20240906095613]$ ./cs.status.aux_cam_gnu_20240906095613 | grep -v PASS | grep -v MEMCOMP | grep -v TPUTCOMP
aux_cam_gnu_20240906095613: 26 tests

NAG test fails one test which I am looking into

[elundgren@izumi aux_cam_nag_20240906085937]$ ./cs.status.aux_cam_nag_20240906085937 | grep -v PASS | grep -v MEMCOMP | grep -v TPUTCOMP
aux_cam_nag_20240906085937: 39 tests
  DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae (Overall: FAIL) details:
    FAIL DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae RUN time=10
    PEND DAE.f45_f45_mg37.FHS94.izumi_nag.cam-dae COMPARE_base_da

@lizziel - If you look at the ChangeLog, you will see that the DAE test is an "expected failure" There are several on derecho/intel as well right now.

[lizziel]

Yes, I just saw from the changelog that the DAE fail is a preexisting failure. I just pushed updates to Changelog for this PR to show izumi test results. @cacraigucar, @fvitt, could you provide more guidance on submitting testDriver on derecho?

[lizziel]

@fvitt, @cacraigucar - It looks like this PR is out-of-date again with latest cam_development. Let me know when you plan to bring this in and I will update to latest cam_development at that time and redo tests. Testing on Izumi worked well but I need more guidance for testing on derecho to do the full test suite.

[fvitt]

@cacraigucar This is ready for review by others.

[lizziel]

Regarding input files, is there a procedure for detecting when new inputs files are added? Should I be creating an issue when there are files I add, or alerting someone via email?

[fvitt]

Regarding input files, is there a procedure for detecting when new inputs files are added? Should I be creating an issue when there are files I add, or alerting someone via email?

@lizziel I have imported the new input files to the inputdata repo this morning.

[cacraigucar]

Regarding input files, is there a procedure for detecting when new inputs files are added? Should I be creating an issue when there are files I add, or alerting someone via email?

It is good to made a note in the issue when new files are added and make sure they are added to the svn repo. That is something I try to check during review, but it is error prone.

Before an official release (like CESM3), the CESM tester runs a special test to check for files not in the repo. Missing files would be found at that time. The only other way they are found, is when a user runs on a machine other than derecho and reports back to us that they can't access the file.

[lizziel]

This PR is now rebased on cam6_4_037 and the changelog is updated.

[fvitt]

@lizziel I have submitted the regression tests. I assume we are expecting bit-for-bit unchanged results for all configurations that are not GEOSCHEM.