GEOS-Chem restart tests fails

[lizziel]

What happened?

This is a known problem that previously did not have a CAM issue dedicated to it. This github issue serves as a reference that this problem exists and to track when it is fixed.

The problem is when breaking up a run using GEOS-Chem chemistry into more than one segment, i.e. CONTINUE_RUN is TRUE after the initial run. This introduces small differences in comparison to a run that does not get broken up in time.

Note that the CAM test suite performed when tagging a new version does not have any GEOS-Chem tests failing. This is because only the SMS test is done for GEOS-Chem. The tests that fail are ERP tests which are available to run manually.

I investigated the problem and there are multiple sources of differences.

1. Several non-concentration arrays need to be included in the restart file. These include GEOS-Chem fields:

   • State_Chm%AeroH2O(:,:,:,NDUST+1)
   • State_Chm%H2O2AfterChem
   • State_Chm%SO2AfterChem
   • State_Chm%KPPHvalue
   • State_Chm%ORVCsesq

2. Differences in the eFlx array returned from subroutine GC_Emissions_Calc (this subroutine is part of the CAM interface in CAM file src/chemistry/geoschem/geoschem_emissions_mod.F90)

3. Differences in concentrations after execution of GEOS-Chem subroutine Do_Chemistry.

I have a branch with updates needed to save and read the non-concentration arrays. See github.com/geoschem/CAM branch bugfix/include_non_conc_GC_arrays_in_restart. More work is needed to determine if additional arrays are needed to include in the restart file to eliminate restart differences, and to determine if there are any bugs contributing to the problem.

What are the steps to reproduce the bug?

Run any of the following GEOS-Chem tests in CAM (category geoschem):

geoschem          : ERP_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s                                                                          
geoschem          : ERP_Ln9.f09_f09_mg17.FC2010climo_GC.derecho_intel.cam-outfrq9s                                                                      
geoschem          : ERP_Ln9.f09_f09_mg17.FC2000climo_GC.derecho_intel.cam-outfrq9s  

What CAM tag were you using?

cam6_4_097

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/u/home/elundgren/code.cam6_add_GC_restart_vars/cime/scripts/case.FC2000climo_GC_double

Will you be addressing this bug yourself?

No

Extra info

No response