Move the dust emission source function soil erodibility for the Zender scheme from CAM to CTSM

[dmleung]

Description of changes We have proposed in #1836 and ESCOMP/CAM#651 to move the dust namelist options in CAM (under group 'dust_nl' in CAM) to CTSM to enable the switch between the two dust emission schemes (per #1897) in the future CTSM versions. We use CTSM's streams capability to read in and interpolate the soil erodibility files. The files were originally read in by CAM from /glade/p/cesmdata/cseg/inputdata/atm/cam/dst/. I have rewritten the files so that CTSM can read them in. For now this branch is capable of reading Zender's erodibility file in CAM6: dst_source2x2tunedcam6-2x2-04062017.nc Later we may probably add in the capability of reading other erodibility files for older versions of CAM (dst_source2x2_cam5.4_c150327.nc, dst_source2x2tuned-cam4-06132012.nc, dst_source1x1tuned-cam4-06202012.nc...) This code also read in the global tuning factor (dust_emis_fact) to tune the global total dust emission. We note that in CAM's convention, all grids of dust emissions are divided by dust_emis_fact instead of multiplied by the factor. So, here in CLM we follow the convention (for now?), and so the larger the factor the smaller the CLM dust emissions.

Changes in files mainly include: src/biogeochem/DUSTMod.F90 src/cpl/share_esmf/ZenderSoilErodStreamType.F90 (new file) src/main/clm_inst.F90 bld/namelist_files/namelist_defaults_ctsm.xml bld/CLMBuildNamelist.pm bld/namelist_files/namelist_definition_ctsm.xml

Specific notes This PR is proposed here and on CAM GitHub (to not read in the dust namelist options in CAM); see ESCOMP/CAM#748. This PR needs to be modified to combine with the recent PR #1897 and ESCOMP/CAM#748.

Contributors other than yourself, if any: @ekluzek

CTSM Issues Fixed (include github issue #): Fixes #2117 Addresses #2149 Addressed part of #1836 Helps with ESCOMP/CAM#651

Are answers expected to change (and if so in what way)? The stream interpolation will possibly slightly change the dust emission flux being used in CAM.

Any User Interface Changes (namelist or namelist defaults changes)? Yes, bld/namelist_files/namelist_definition_ctsm.xml bld/namelist_files/namelist_defaults_ctsm.xml bld/CLMBuildNamelist.pm

[wwieder]

Awesome to see this PR coming in, Danny. I'm sure we'll be in touch as we review this code and as you continue to develop / finalize the code

Thanks for these contributions to CESM!

[dmleung]

Thanks Will! Keep in touch :)

[dmleung]

Hi @fvitt, @ekluzek and I would like to ask you a question:

• We noticed there are several soil erodibility files in the namelist_defaults_cam.xml for CAM4, CAM5, and CAM6. Should we move all of them to CTSM or should we just take the CAM6 one? Do you think people can actually use CAM4 but still have CTSM using the CAM6 erodibility file to calculate dust emissions?
• I am planning to copy all default values of the global tuning factor (dust_emis_fact) from namelist_defaults_cam.xml to namelist_defaults_ctsm.xml (maybe divided by 1.15). Does that sound good to you?

[fvitt]

@dmleung I don't know which soil erod files should be moved to CTSM. @lkemmons and @tilmes may have an opinion on this.

[tilmes]

I think, that there are people out there that still use different model configurations, like CAM4/WACCM4 (like Chuck for CARMA), and maybe CAM5 (I am not sure). I am not sure if people can run CAM4 using the CAM6 erodibility file, I don't know how those files differ from each other.

[dmleung]

Okay, thank you @fvitt and @tilmes for the reply. I don't think the erod files used across different CAM versions are that different (the values differ slightly). I will look at the erod files again, think about it and discuss with people further. @L3atm, please feel free to add any thoughts if helpful.

[ekluzek]

One thing to add here is the global dust tuning parameter to the CTSM namelist.

[ekluzek]

@dmleung Louisa @tilmes @wwieder @fvitt and I met to discuss this. We decided:

• The global dust tuning parameter dust_emis_fact will remain in CAM (although it's value will be expanded to take into account the dust emission method)
• We will make the CMEPS and then CTSM first, so this will all work together.

The issue about the dust emission method for CMEPS is now here:

https://github.com/ESCOMP/CMEPS/issues/353

[ekluzek]

I've added a little design document about how to do this in CMEPS here:

https://docs.google.com/document/d/18nZ3LJF5W-YF9iBhqed6s_NWeKOvSSL2-k0Lye1nnLg

[ekluzek]

The design doc for the CMEPS shr_ part of this is in:

https://docs.google.com/document/d/18nZ3LJF5W-YF9iBhqed6s_NWeKOvSSL2-k0Lye1nnLg

[ekluzek]

I met with @dmleung and went over the current plans. I had proposed that CAM4, and CAM5 soil erodibility files be kept in CAM because of the effort needed to convert them to use in CTSM with streams. But, then he pointed out that he had already done this. So the files are already converted.

Here is the list of files in CAM right now...

<!-- soil erodibility factors -->
<soil_erod_file              >atm/cam/dst/dst_source2x2tunedcam6-2x2-04062017.nc </soil_erod_file>
<soil_erod_file phys="cam5"  >atm/cam/dst/dst_source2x2_cam5.4_c150327.nc</soil_erod_file>
<soil_erod_file phys="cam4"  >atm/cam/dst/dst_source2x2tuned-cam4-06132012.nc</soil_erod_file>
<soil_erod_file phys="cam4" hgrid="0.9x1.25">atm/cam/dst/dst_source1x1tuned-cam4-06202012.nc</soil_erod_file>

I can move these to CTSM, but need to use LND_TUNING_MODE to set whether CAM4, CAM5, or other version of CAM physics is being used. We currently don't have LND_TUNING_MODE settings for CAM4 or CAM5, but I can add them in. I will at the same time have to add those settings for the other fields that are tuned for CTSM, but will base them on the currently tuned values.

[dmleung]

@ekluzek We can do either one of them, depending on which approach is easier for you and for the model. For record, the path to the converted soil erodibility files are in /gpfs/fs1/work/dleung/Zender_dust_source

The stream file, src/cpl/share_esmf/ZenderSoilErodStreamType.F90, can process all soil erodibility files using the namelist variable stream_fldfilename_zendersoilerod. I think we only need to add more entries in the bld/namelist_files/namelist_defaults_ctsm.xml. Currently it contains the path to the CAM6 soil erodibility file

<stream_fldfilename_zendersoilerod /gpfs/fs1/work/dleung/Zender_dust_source/cdf5_dst_source2x2tunedcam6-2x2-forCLM_c230311.nc</stream_fldfilename_zendersoilerod>

But we can add more entries for CTSM to use CAM5 and CAM4 soil erodibility files by detecting the CAM version (or physics) in an F case.

[ekluzek]

In order to facilitate getting the major thrust of this in, and to not tie this work to a CMEPS external, I will break this up into two bits of work. This bit will come in largely as is, with just an additional control variable added to the CTSM namelist to control if the soil erodibility file should be used here in CTSM. A future simple PR will bring in the CMEPS external update and remove the newly added CTSM namelist item.

By breaking it up this way this becomes a smaller project that is mostly done. Here are tasks remaining:

1. [x] Resolve some questions I have on how to handle LND_TUNING_MODE 1
2. [x] Handle the namelist part in build-namelist and namelist XML, get the namelist testing to work, add tests for it 3-5
3. [x] Handle the namelist part in FORTRAN 4-6
4. [x] Put the control in the code where needed 2-4
5. [x] Move the new files into CESM inputdata on glade and rimport 1
6. [x] Add tests that exercise this to the testmods and testlist 2
7. [x] Run testing as is in ctsm5.1.dev174-b4b-dev, resolve issues 4-8
8. [x] Bring in for b4b-dev Respond to reviews 1-2
9. [x] Update to the latest version (I tried updating to ctsm5.1.dev139 and this is expected to be simple with only one conflict) 2
10. [x] Run testing again 1-4
11. [x] Merge to b4b-dev 1

My time estimate for this part in terms of hours is:

Min: 22 hours Avg: 36 hours Max: 42 hours

So estimate is average of above at: 33 hours

[ekluzek]

This is about ready to come in now. Almost all tests pass, with the exception of:

ERP_D_Ld9.ne30pg3_t232.I1850Clm51BgcCrop.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod

which fails inexplicably. And

ERI_D.ne30pg3_t232.I1850Clm51BgcCrop.derecho_intel.clm-clm51cam6LndTuningMode

which says it has different answers than the baseline, but I totally don't get why that's the case.

[ekluzek]

I reran the ERI test from scratch and I'm still getting a difference to baseline which is confusing to me.

[ekluzek]

I also realized that the problem with the other tests seems to be that using the Zender soil-eroditablity files bumps up the memory usage just enough that tests fail. But, by using more processors the tests start passing. I also tried more resolutions and found many other resolutions fail because of this.

So this is really due to the work we need to do in #2244

Here's some info on the tests I ran:

================================================================================
These tests passed
================================================================================
ERI_D.ne30pg3_t232.I1850Clm51BgcCrop.derecho_intel.clm-clm51cam6LndTuningMode			
SMS_D_Ln9.C96_C96_mt061.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_D_Ln9.ne3pg3_ne3pg3_mg37.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_P1024x1_D_Ln9.f19_f19_mg17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_P2048x1_D_Ln9.ne16pg3_ne16pg3_mg17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_P512x1_D_Ln9.f45_f45_mg37.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_P512x1_D_Ln9.mpasa480_mpasa480.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
SMS_P6144x1_D_Ln9.C96_C96_mt061.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod			
================================================================================
These tests are pending (some tests may fail in the pending state)
================================================================================
SMS_D_P3072x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
================================================================================
These tests failed
================================================================================
SMS_D_Ln9.f19_f19_g17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_D_Ln9.f45_f45_mg37.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_D_Ln9.mpasa480_mpasa480.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_D_Ln9.ne16pg3_ne16pg3_mg17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_D_P384x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_D_P768x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod				
SMS_P512x1_D_Ln9.f19_f19_g17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod		
SMS_P512x1_D_Ln9.ne16pg3_ne16pg3_mg17.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod

[ekluzek]

OK some strange things about testing with ne30 grids. If I run with too few processors it will fail with what I think are memory issues. With enough though it scales backwards so that more processors takes more overall run time, and 24 nodes causes it to hang. This is looking at total run time in the TestStatus file, so not accurate, but here it is...

SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:PASS SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=85 SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:PASS SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=105 SMS_D_P3072x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:FAIL SMS_D_P3072x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=13275 (this hangs)

This is validated with the timing file, for model throughput. But, probably more testing needs to be done...

SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/timing/cesm_timing.SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist.3607734.desched1.240228-152003: Model Throughput: 6.40 simulated_years/day SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/timing/cesm_timing.SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist.3684538.desched1.240301-155809: Model Throughput: 5.59 simulated_years/day

[ekluzek]

I thought maybe the problem with the ERI test was that the baselines had been messed up somehow. So I reran the baseline from ctsm5.1.dev169 and expected it to differ from the existing baselines. However, it compared exactly and showed no problems.

The difference in the test does seem like it's comparing different parts of the testing which would make it fail. But, I probably need to do more work to track down what it's doing.

Since, other tests are showing identical results. Including these ERI tests...

ERI_C2_Ld9.f10_f10_mg37.I2000Clm51BgcCrop.derecho_gnu.clm-default ERI_D.ne30pg3_t232.I1850Clm51BgcCrop.derecho_intel.clm-clm51cam6LndTuningMode ERI_D_Ld9.f10_f10_mg37.I1850Clm45Bgc.derecho_gnu.clm-default ERI_D_Ld9.f10_f10_mg37.I1850Clm51Bgc.derecho_gnu.clm-default ERI_D_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-default ERI_D_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-default ERI_D_Ld9.ne30_g17.I2000Clm50BgcCru.derecho_intel.clm-vrtlay ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-default ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-drydepnomegan ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-default ERI_Ld9.f45_g37.I2000Clm50BgcCru.derecho_intel.clm-nofire

I'm also wondering if this is something that we can ignore as an odd glitch...

[ekluzek]

I was having trouble with the following test on izumi:

SMS_Ln9.f10_f10_mg37.I2000Clm45Sp.izumi_gnu.clm-clm45cam4LndTuningModeZDustSoilErod

It turned out that running with NAG found a legit problem!

diff --git a/src/cpl/share_esmf/ZenderSoilErodStreamType.F90 b/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
index 1755a32f7..0d5a7653d 100644
--- a/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
+++ b/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
@@ -90,8 +90,8 @@ subroutine Init(this, bounds, NLFilename)
    character(len=*), parameter    :: stream_name = 'zendersoilerod'
    !-----------------------------------------------------------------------
 
-      call this%InitAllocate( bounds )
       call control%ReadNML( bounds, NLFileName )
+      call this%InitAllocate( bounds )
 
       if ( this%useStreams() )then     ! is this a namelist input and is it set in namelist default

Without this change an array doesn't get properly allocated! So this was a critical fix and it only showed up with the Nag compiler on izumi!

The cesm.log file showed the following:

Runtime Error: /fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/share_esmf/ZenderSoilErodStreamType.F90, line 160: Subscript 1 of THIS%SOIL_ERODIBILITY (value 1) is out of range (1:0)
Program terminated by fatal error
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/share_esmf/ZenderSoilErodStreamType.F90, line 160: Error occurred in ZENDERSOILERODSTREAMTYPE:INIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/biogeochem/DUSTMod.F90, line 90: Called by DUSTMOD:INIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/main/clm_instMod.F90, line 353: Called by CLM_INSTMOD:CLM_INSTINIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/main/clm_initializeMod.F90, line 406: Called by CLM_INITIALIZEMOD:INITIALIZE2
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/nuopc/lnd_comp_nuopc.F90, line 659: Called by LND_COMP_NUOPC:INITIALIZEREALIZE
/fs/cgd/data0/erik/ctsm_worktree/zendersource/components/cmeps/cime_config/../cesm/driver/esmApp.F90, line 128: Called by ESMAPP

@briandobbins this is an example of how using the Nag compiler can help us with our code.

[ekluzek]

I realized the continuing to fail ERI test is because of the finidat/use_init_interp settings that apparently was only showing up for this one test. Some changes in namelist_defaults_ctsm.xml should allow it to work.