Fan 2019 ctsm

[juliusvira]

Description of changes

Includes the FANv2 (Flow of Agricultural Nitrogen) process model for simulating ammonia volatilization from fertilizers and manure.

Specific notes

Are answers expected to change (and if so in what way)?

Connecting FAN to the CLM biogeochemistry (nitrogen pools) can be toggled from namelist. If not connected, answers should not change. If connected, C and N cycling will change.

Any User Interface Changes (namelist or namelist defaults changes)?

Added FAN namelist with some options and path to the FAN stream file.

Testing performed, if any:

The model has been tested with a 3-year (2009-2011) transient run for the following:

• C and N related variables do not change if FAN is off or not coupled to the biogeochemistry
• FAN produces NH3 emissions as expected
• Running with biogeochemistry coupling enabled works technically and conserves N. Scientific evaluation of the biogeochemistry coupling is to be done.

Other remarks:

• FAN has the capability to move N from crop to soil column within a grid cell, depending on the fract_spread_grass namelist variable.
• The N balance check currently essentially checks separately for N conservation in FAN and in the rest of CLM. The advantage is that this works the same way regardless of whether FAN is coupled to the biogeochemistry or not. The drawback is that mass conservation between FAN and CLM is not checked. There could be a few ways to fix this.

Edit: link to slides comparing various FAN coupling options: https://drive.google.com/file/d/1CROPPXphlwhWCMbLKl_RKUaaSehUomHg/view?usp=sharing

[ekluzek]

@juliusvira can you add me as a collaborator to your fork of ctsm, so I can push to your branch? Thanks.

[ekluzek]

@juliusvira actually I need collaborator access to your cime branch as well. You go to settings on your fork, and then collaborators, and add @ekluzek with push access.

[ekluzek]

OK, so I've updated to ctsm1.0.dev065 and ran testing on cheyenne. There are several issues coming out. With FAN totally off, answers are different for NFERTILIZATION for the following tests:

ERI_N2_Ld9.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-default ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart ERP_D_Ld5.f19_g17_gl4.I1850Clm50BgcCrop.cheyenne_intel.clm-glcMEC_changeFlags ERP_D_P36x2_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_gnu.clm-default ERP_D_P36x2_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default ERP_D_P36x2_Ld3.f10_f10_musgs.I2000Clm45BgcCrop.cheyenne_gnu.clm-no_subgrid_fluxes ERP_D_P36x2_Ld5.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-irrig_spunup ERP_D_P36x2_Ld5.f10_f10_musgs.I2000Clm50BgcCropRtm.cheyenne_intel.clm-irrig_spunup ERP_Ld3.f09_g17.I1850Clm50BgcCropCru.cheyenne_intel.clm-ciso ERP_Ly3_P72x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-cropMonthOutput ERP_P180x2_D_Ld5.f19_g17_gl4.I1850Clm50BgcCropG.cheyenne_intel.clm-glcMEC_increase ERP_P36x2_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_gnu.clm-extra_outputs ERP_P36x2_D_Ld5.f10_f10_musgs.I2000Ctsm50NwpBgcCropGswpGs.cheyenne_intel.clm-default ERP_P72x2_Lm25.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-monthly ERP_P72x2_Lm36.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-clm50cropIrrigMonth_interp ERP_P72x2_Lm7.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-irrig_alternate_monthly ERP_P72x2_Ly3.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-irrig_o3_reduceOutput ERS_D.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-reseedresetsnow ERS_D_Ld12.f10_f10_musgs.I1850Clm50BgcCropG.cheyenne_intel.clm-glcMEC_spunup_inc_dec_bgc ERS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_gnu.clm-default ERS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default ERS_D_Ld3.f19_g17_gl4.I1850Clm50BgcCrop.cheyenne_intel.clm-clm50dynroots ERS_Lm40_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianGs.cheyenne_gnu.clm-monthly ERS_Ly3.f10_f10_musgs.I1850Clm50BgcCropCmip6.cheyenne_intel.clm-basic ERS_Ly3_P72x2.f10_f10_musgs.IHistClm50BgcCropG.cheyenne_intel.clm-cropMonthOutput ERS_Ly5_P144x1.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-cropMonthOutput ERS_Ly5_P72x1.f10_f10_musgs.IHistClm45BgcCrop.cheyenne_intel.clm-cropMonthOutput LCISO_Lm13.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_monthly LII2FINIDATAREAS_D_P360x2_Ld1.f09_g17.I1850Clm50BgcCrop.cheyenne_intel.clm-default LII_D_Ld3.f19_g17_gl4.I2000Clm50BgcCrop.cheyenne_intel.clm-glcMEC_spunup_1way PET_P36x2_D.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_gnu.clm-default PET_P36x2_D.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default SMS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianGs.cheyenne_intel.clm-cropMonthOutput SMS_Lm1.f10_f10_musgs.I1850Clm50BgcCropCmip6waccm.cheyenne_gnu.clm-basic SMS_Lm13.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-cropMonthOutput SMS_Lm1_D.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-output_crop_highfreq SMS_Lm1_D.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-snowlayers_3_monthly SMS_Ly3_Mmpi-serial.1x1_numaIA.I2000Clm50BgcDvCropQianGs.cheyenne_gnu.clm-ignor_warn_cropMonthOutput SOILSTRUCTUD_Ld5.f10_f10_musgs.I2000Clm50BgcCropGs.cheyenne_intel.clm-default SSP_D_Ld4.f09_g17.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_rtmColdSSP

Threaded tests fail:

ERP_D_P36x2_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start ERP_D_P36x2_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_intel.clm-fanFull2009Start

This test seems to take too long...

SMS_Lm9.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start

Tests with the newer land/ocean mask f19_f19_mg17 fail (for example: SMS_Ld1.f19_f19_mg17.IHistClm50BgcCropGs.cheyenne_intel.clm-fanFull).

[ekluzek]

Looks like there is some extra log output that should be backed off. And it looks like FAN is slow and needs to be looked at for optimization (I keep having to increase the time limit on some of the fan tests).

[ekluzek]

Non-threaded ERP tests do pass

ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start

PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start CREATE_NEWCASE PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start XML PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start SETUP PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start SHAREDLIB_BUILD time=112 PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start MODEL_BUILD time=43 PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start SUBMIT PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start RUN time=3174 PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start COMPARE_base_rest PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start MEMLEAK insuffiencient data for memleak test PASS ERP_D_Ld3.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start SHORT_TERM_ARCHIVER

[juliusvira]

I think that the NFERTILIZATION is different because in the fan branch it had to be split to fertilizer and manure N in NMANURE. If you want it to be backwards compatible, then with a bit of code, it would possible to put the sum in NFERTILIZER and make fan subtract one from another.

[ekluzek]

@juliusvira ahh good. So NFERTILIZATION was the sum of commercial and manure fertilization. And with FAN it becomes split into NFERTILIZATION and NMANURE. So if you keep NFERTILIZATION as the sum, you'll still be able to get just the commercial component by subtracting out NMANURE. So you can still get the two quantities however, you like as long as you have both NMANURE and NFERTILIZATION being the sum of commercial plus NMANURE.

That would make it worthwhile doing, just so it's easy to see that the answers are identical with FAN on and off. If we want to change the definition of NFERTILIZATION, we should do that as a separate tag.

So yes, can you go ahead and make that change? Thanks.

[ekluzek]

I gave SMS_Lm9.f19_f19_mg16.IHistClm50BgcCropGs.cheyenne_gnu.clm-fanFull2009Start, 3 hours and 40 minutes to run, and it still didn't finish, needing about 9X that to run. So especially with the gnu compiler it's running really slow and we need to look into why.

[juliusvira]

The FAN runs I did were not obviously slower than the ones without it, and certainly not 9X slower. That was with the intel compiler. However, I use gfortran in the Cornell servers and the cpu-hours/year there doesn't seem that different from Cheyenne.

[ekluzek]

OK, good. It might just be something about gnu. I did see that there's a bunch of extra log output, and it could be that gnu is just extra sensitive to that. Or just extra sensitive to something else like that. So it might be easily fixed.

[wwieder]

Let's set NMANURE to zero (and set of off by default) when NMANURE is being ignored (BGCfull) and create new history fields for the NMANURE_FAN (N manure inputs to the FAN model) and NMANURE_FAN_TOSMINN (N manure inputs leaving FAN and going to the soil bgc). @juliusvira can your propose a list of variables we'll want and their definition?

[juliusvira]

The N inputs to FAN are

• FERT_N_APP = NFERTILIZATION mapped to columns
• MAN_N_BARNS in crop columns
• MAN_N_GRZ in native veg. columns. The total manure production is always MAN_N_BARNS + MAN_N_GRZ on grid cell level and also on column level if fract_spread_grass is 0. If fract_spread_grass > 0, it gets more complicated since some of MAN_N_BARNS is rerouted to the native column.

The total loss from FAN pools is NH3_TOTAL + MANURE_NH4_RUNOFF + FERT_NH4_RUNOFF. I don't know if we need a new field to add these up. The fluxes that can go to FERT_TO_SMINN are MANURE_NH4_TO_SOIL, FERT_NH4_TO_SOIL, MANURE_NO3_PROD and FERT_NO3_PROD.

It seemed to me that you'll want something like

• MAN_N_TOTAL = MAN_N_BARNS + MAN_N_GRZ
• MAN_N_TO_SMINN = MANURE_NH4_TO_SOIL+MANURE_NO3_PROD
• SYNTHFERT_N_TO_SMINN = FERT_NH4_TO_SOIL + FERT_NO3_PROD or whatever names sound good.

I think that MAN_N_TO_SMINN and SYNTHFERT_N_TO_SMINN should depend on the FAN coupling mode so that they are 0 for the non-coupled columns. Then we'd have MAN_N_TO_SMINN+SYNTHFERT_N_TO_SMINN = FERT_TO_SMINN in either of the bgc-coupled modes and MAN_N_TO_SMINN and SYNTHFERT_N_TO_SMINN = 0 if FAN is not coupled.

[wwieder]

@juliusvira I think this is making sense. I have a few clarifications based on your post, above. The first two may be more important to resolve, given @ekluzek comments previously

• [ ] NFERTILIZATION should always be the sum of manure and synthetic fertilizer inputs
• With FANoff this should be unchanged, the sum of parameter file manure and streams N inputs, Note, this is a patch level calculation
• With FANon we have two manure inputs being used. One from the parameter file that's still being passed to crops and a second that is being read in from FAN's dataset and ONLY used to calculate HN3 emissions and NO3 leaching. What manure do we add to NFERTILIZATION here, it seems we should just keep track of the one that crops actually see?
• With FANbgc this would be NFERTILIZATION = MANURE_N_TOTAL + SYNTHFERT_N_TOTAL This is a column level calculation

For summing manure and synthetic fertilizer inputs to NFERTILIZATION, maybe it makes the most sense to calculate this in the fan_to_sminn subroutine in Fan2CTSMMod.F90, which would need to be called even for FANoff.

In all cases, SYNTHFERT_N_TOTAL would be the same as what's defined on the input streams file Hopefully this gets around @ekluzek note that without FAN, "NFERTILIZATION was the sum of commercial and manure fertilization". The suggestion above makes NFERTILIZATION defined consistently, and avoids creating new history files that can be confusing.

For clarity I'd also suggest the following field:

• [ ] NFERTILIZATION_TO_SMINN = MANURE_N_TO_SMINN + SYNTHFERT_N_TO_SMINN These would only be turned on for FANbgc simulations AND/OR Could be defined differently NFERTILIZATION_TO_SMINN = NFERTILIZATION For FANoff or FANon simulations (and not connected to soilBGC) As with NFERTILIZATION, this should be calculated this in the fan_to_sminn subroutine of Fan2CTSMMod.F90

• [ ] A minor suggestion would be to write out MANURE (as MAN is confusing for other model users) MANURE_N_TOTAL = MANURE_N_BARNS + MANURE_N_GRZ MANURE_N_TO_SMINN = MANURE_NH4_TO_SOIL+MANURE_NO3_TO_SOIL SYNTHFERT_N_TO_SMINN = SYNTHFERT_NH4_TO_SOIL + SYNTHFERT_NO3_TO_SOIL

• [ ] Last detail. Why the distinction between _NH4_TO_SOIL vs. _NO3_PROD in your comment @juliusvira ? Is the assumption currently applied that all the NO3 losses from FAN interact with the soil matrix, or are some of these LEACHED terms supposed to go straight to runoff and the rivers? It seems these _SMINN fluxes should just be the fraction that's going to the soil BGC? SYNTHFERT_N_TO_SMINN = SYNTHFERT_NH4_TO_SOIL + SYNTHFERT_NO3_TO_SOIL in other words, does SYNTHFERT_NO3_TO_SOIL == SYNTHFERT_NO3_PROD?

Is this seeming more clear?

[ekluzek]

Performance with FAN on seems to be fine with the intel compiler, but dog slow for at least gnu and nag compilers (gnu is running at 0.49 sim-years/compute-day). From looking at the timing file, it doesn't look like it's actually FAN itself, but history output? And also maybe the diagonstic write (wrtdia)?

[ekluzek]

I'm getting FAN to crash when I try to run for a 2000-control case (as opposed to a Historical case that runs from 2009-2011). Here's a simple way to test that and the results I got...

cd cime/scripts
qcmd -- ./create_test SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull -r .

Which results in...

PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull CREATE_NEWCASE PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull XML PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull SETUP PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull SHAREDLIB_BUILD time=229 PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull MODEL_BUILD time=203 PASS SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull SUBMIT FAIL SMS_Ld1.f10_f10_musgs.I2000Clm50BgcCrop.cheyenne_intel.clm-fanFull RUN time=40

Here's the output...

20: SoilPH check:   6.22678995132446        7.50562763214111                0
27: bad flux_avail                     NaN                     NaN
27:  0.181500686073526     
27: ENDRUN:
27: ERROR: bad flux_avail
34: Balance check:Manure  3.428601433175966E-003  2.280753313301468E-003
34: Balance check:Fertilizer  0.000000000000000E+000  0.000000000000000E+000
34: SoilPH check:   6.72264099121094        6.84946060180664                0
67: WARNING: qice out of bounds: g, n, qice =         264           1
67:  -13.5012076451853     
37:Image              PC                Routine            Line        Source             
37:cesm.exe           0000000001A6737A  Unknown               Unknown  Unknown
37:cesm.exe           00000000011C9740  shr_abort_mod_mp_         114  shr_abort_mod.F90
37:cesm.exe           000000000050396F  abortutils_mp_end          50  abortutils.F90
37:cesm.exe           00000000007AE59F  fan2ctsmmod_mp_ha         891  Fan2CTSMMod.F90
37:cesm.exe           00000000007A591F  fan2ctsmmod_mp_fa         321  Fan2CTSMMod.F90
37:cesm.exe           00000000006E1AF4  cnndynamicsmod_mp         184  CNNDynamicsMod.F90
37:cesm.exe           0000000000BCCBD6  cndrivermod_mp_cn         272  CNDriverMod.F90
37:cesm.exe           0000000000714708  cnvegetationfacad         849  CNVegetationFacade.F90
37:cesm.exe           000000000051018B  clm_driver_mp_clm         890  clm_driver.F90
37:cesm.exe           00000000004F8B13  lnd_comp_mct_mp_l         458  lnd_comp_mct.F90
37:cesm.exe           0000000000429234  component_mod_mp_         737  component_mod.F90
37:cesm.exe           000000000040A13B  cime_comp_mod_mp_        2606  cime_comp_mod.F90
37:cesm.exe           0000000000428E6C  MAIN__                    133  cime_driver.F90
37:cesm.exe           0000000000408022  Unknown               Unknown  Unknown
37:libc.so.6          00002B63763496E5  __libc_start_main     Unknown  Unknown
37:cesm.exe           0000000000407F29  Unknown               Unknown  Unknown

[juliusvira]

Based on the previous discussions I've pushed a set of changes with new/newly defined output variables:

• MANURE_N_TOTAL = MAN_N_BARNS + MAN_N_GRZ whenever fan is on
• NFERTILIZATION is the N that be would go to the soil mineral pools in absence of losses: NFERTILIZATION = NSYNTHFERT+NMANURE (fan off or not coupled to bgc) NFERTILIZATION = MANURE_N_BARNS+FERT_N_APP (in fanbgc-crop if fract_spread_grass = 0) NFERTILIZATION = MANURE_N_TOTAL+FERT_N_APP (in fanbgc-full) where FERT_N_APP = NSYNTHFERT, but only created when fan is on.
• MANURE_N_TO_SMINN and SYNTHFERT_N_TO_SMINN are the FAN fluxes to the soil N pools after losses; ie. SYNTHFERT_N_TO_SMINN = FERT_NH4_TO_SOIL + FERT_NO3_TO_SOIL and similar for manure. If the bgc coupling is not active ("fanon") then these are zero.

NMANURE is zero in the bgc-coupled fan modes. NSYNTHFERT stays same. I changed MAN_ to MANURE_ and _NO3_PROD to _NO3_TO_SOIL. I finally removed the man_n_stored and man_tan_stored fields because FAN does actually not use them.

Hopefully I interpreted everything right...

[ekluzek]

@juliusvira I'm not seeing those changes. Are you sure you pushed to your "fan-2019-ctsm" branch on your juliusvira fork?

[juliusvira]

@juliusvira I'm not seeing those changes. Are you sure you pushed to your "fan-2019-ctsm" branch on your juliusvira fork?

I do think so (https://github.com/juliusvira/ctsm/commits/fan-2019-ctsm) -- unless I'm missing something. Github shows above everything to be "17 days ago" but some of the commits are newer.

I forgot to mention the new stream files: for the three resolutions, they are at /glade/work/juliusv/nitrogen/fan_nitrogen_{f19,f09,f05}_c20191009.nc

[ekluzek]

OK, I see the changes now. Thanks for checking. It seems to show the commit date, but not the push date. But, anyway it's there now.

On the datasets, do we really need to support three different resolutions for the manure streams datasets? We would prefer to just have it at half degree resolution and use the streams bi-linear interpolation all the time. That makes the dataset management easier.

[wwieder]

• @ekluzek this isn't a priority for CESM2.2 of PPE tags, but I think it should be included in the CTSM5.1 release.
• @juliusvira one thing we'll need is a description from you for the technical note. There's a bunch of documentation on how to do this at the bottom of the wiki page (like this).
• There are a handful of items still marked with a check box, I'll go through and see if these are resolved.

[ekluzek]

Some issues we thought of about this have to do with the use of streams in this PR. As implemented each variable is done with a separate stream, and it should just have one for all streams.

The other issue is that NUOPC streams are different than MCT streams, so it would need to be updated to the NUOPC form.

[wwieder]

@danicalombardozzi just told me that she's part of a team with Peter Hess that has a newly funded project to simulate NOx emissions from soils that will build on this work. Bringing this PR to main will make future model developments much easier. Can we move this PR back up the priority list?

[juliusvira]

@wwieder would you say that the most important missing piece is the tech note? I can try to find time to work with it during this spring. Should the tech note replicate the contents of the GMD paper and its supplement, or should refer to those in the note? I'm trying to understand the level of detail that is expected.

I can still handle the code cleanup items above. Is @ekluzek able to deal with the necessary changes to the setup scripts?

[wwieder]

Technical documentation will be critical @juliusvira , but more critical will be finding time for @ekluzek to handle more technical aspects of the SE work that are required to bring this to main.

[wwieder]

@ekluzek now that you've created a branch tag for this PR, does it remain open?

[ekluzek]

I think we leave it open as a reminder that we eventually want to bring it into main CTSM. But as we have plenty of open PRs we could close it for now. Let's talk about this Thursday.