PLUMBER2 plumbing (csv file, wrapper script, usermods, and scripts).

[olyson]

Description of changes

This PR adds a csv file for PLUMBER2 tower sites in preparation for incorporating PLUMBER2 into the NEON framework. The ncl script that generates this file is included as well

Specific notes

These are added to tools/site_and_regional for now.

Contributors other than yourself, if any: @ekluzek @wwieder @slevis-lmwg @TeaganKing @adrifoster

CTSM Issues Fixed (include github issue #): Partial for https://github.com/ESCOMP/CTSM/issues/2137

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Testing performed, if any: None

[ekluzek]

I added one more item to do before coming in, add a system test for a PLUMBER2 site, so we are at least testing one of the PLUMBER2 user-mods and are more likely to find issues that happen as the code changes.

[wwieder]

@ekluzek I could use some help with setting up datm.streams for the PLUMBER cases. I think we'd ideally use streams that span the hist-ssp timeframe as most start dates are before 2014 and a number of sites have end dates that end in 2017 (and eventually later). Maybe this is something we can discuss next week as you wrap your head around what else needs to happen here. Thanks

/glade/u/home/wwieder/scratch/PLUMBER2_tests/ZM-Mon_test2/user_nl_datm_streams contains a streamname 
'presndep.SSP3-7.0' that is not part of valid streamnames ['CLMGSWP3v1.Solar', 'CLMGSWP3v1.Precip', 
'CLMGSWP3v1.TPQW', 'presaero.SSP3-7.0', 'presndep.hist', 'preso3.hist', 'topo.observed', 'co2tseries.20tr'] 

[wwieder]

Incremental progress here. my working case directory is still /glade/scratch/wwieder/PLUMBER2_tests/ZM-Mon_test2

Now I'm pointing to what should be the correct datm file:

<datafiles>
      <file>/glade/work/oleson/PLUMBER2/datm_files/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc</file>

But getting the following dt error that I don't know how to address? @olyson how did you do this previously for PLUMBER2 sites? (shr_stream_findBounds) ERROR: limit on and rDateIn lt rDatelvd 20071231.9166667 20080101.0000000
ERROR: (shr_stream_findBounds) ERROR: rDateIn lt rDatelvd limit true

[olyson]

To spinup towers with only one year of forcing, we had to increase dtlimit. The problem is that because we start at UTC corresponding to local midnight and because when the datm reaches the end of the data it does not cycle back to UTC corresponding to local midnight but to UTC midnight, there is a gap in the forcing data between the end of the data and the start of the next year when looping back over the data. The consequence with the dtlimit approach is that the forcing for UTC midnight is repeated for the number of time steps that local time is different from UTC. For this particular site, that would be four time steps. Decided this isn't that big of a deal for spinup since the forcing is slowly varying. An alternative approach I considered later would be to run a number of startup simulations equal to the number of desired spinup years and initialize each simulation with the initial file created at the end of each previous simulation, which would require some additional logic.

[wwieder]

Sorry to be kind of slow on this, Keith, but is something like this what you mean for user_nl_datm_streams?

CLM_USRDAT.PLUMBER2:datafiles = /glade/work/oleson/PLUMBER2/datm_files/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc
CLM_USRDAT.PLUMBER2:dtlimit=30

I tried setting dtlimit to 4 (as you suggested above) and 30 (as with other datm streams), but both fail. Are units for dtlimit supposed to be in seconds (e.g. 14400?, this also doesn't work)... Is there something else I should be trying here?

[olyson]

It's the ratio of ratio of max/min stream delta times. I'm not as familiar with the new datm streams method so it might be easiest if you point me to your case.

[wwieder]

/glade/scratch/wwieder/PLUMBER2_tests/ZM-Mon_test2/ Thanks in advance

[olyson]

I see from what I think is your case (ZM-Mon_test2) that you are getting this error at the beginning of the simulation. The error I referred to occurs at the end of the first year, so something is different. What happens if you change tintalgo to "nearest" instead of "linear".

[wwieder]

adding CLM_USRDAT.PLUMBER2:tintalgo=nearest to user_nl_datm does not fix this issue. For what it's worth tintalgo=linear for NEON cases.

[wwieder]

I also realized that we're setting the following in env_run.xml

    <entry id="RUN_STARTDATE" value="2007-12-31">
    <entry id="START_TOD" value="79200">

Setting CLM_USRDAT.PLUMBER2:taxmode=cycle generates an new error now, so I'm actually reading in the datm data!

 (datm_datamode_clmncep_advance): tbotmax =    293.891003417969
 (datm_datamode_clmncep_advance): anidrmax =   1.000000000000000E+030
 ERROR: (datm_datamode_clmncep_advance): ERROR: cannot compute rain and snow

[olyson]

I think Sa_precn needs to be changed to Faxa_precn?

[wwieder]

Good spot, Keith.

Now I'm able to run for a few time steps, but the model fails without helpful errors (that I can spot) beyond this in my cesm.log forrtl: severe (61): format/variable-type mismatch, unit -132, file /glade/u/home/wwieder/scratch/PLUMBER2_tests/ZM-Mon_test2/run/atm.log.4030208.chadmin1.ib0.cheyenne.ucar.edu.231101-092559

[olyson]

The line the traceback is pointing to is this: write(sdat%stream(1)%logunit,'(a,i8)') trim(subname),' ERROR: for stream ',ns

and the error "format/variable-type mismatch" makes me think that ns is not an integer (i8), which I don't understand. However, the if statment above it:

if ((sdat%pstrm(ns)%dtmax/sdat%pstrm(ns)%dtmin) > sdat%stream(ns)%dtlimit) then

seems to indicate that the model is trying to identify a dtlimit error. So now try using something large like 50 for dtlimit to see if that continues the run.

[wwieder]

YES! Thanks Keith.

[wwieder]

@olyson helped me get things to work here. I'll push updates and pass this over to you, @ekluzek

[wwieder]

@ekluzek I know you have a lot going on, but I think that some of @TeaganKing's work with #2259 will be facilitated by merging this PR, especially now that we have #2156 on main. I've tried to address some of the check boxes you added above, others I have questions on, and some need help from you or @slevis-lmwg . How should be best proceed?

[ekluzek]

@wwieder good question. I'm going to plan on including it with the Derecho tag, unless it delays getting that in to much. I think this might be close enough that it won't delay much...

[wwieder]

Hi @ekluzek I wanted to circle back on this, because @TeaganKing needs this to continue on her continued work with PLUMBER.

[wwieder]

Hi everyone,

I wondered if we could circle back to this, as it would be helpful to:

• put surface datasets and datm files in the correct place on input data.
• should surface data be updated to CTSM5.2?
• make other adjustments so this works on Derecho.

Happy to help with as much as I can, but need help with the input data issues, @ekluzek or @slevis-lmwg ?

[TeaganKing]

Hi all, I'm wondering what the timeline for wrapping up this PR might be?

I'm working on #1487 and have done some initial refactoring of NeonSite into a neon-specific class as well as a more general tower class that will be the basis for the PLUMBER2 class (#2315 and #2363). It would be ideal if this PR were merged in before getting into the weeds of adding PLUMBER capabilities.

[wwieder]

Yes, I think we can move this up the priority list.

@ekluzek has some check boxes listed above. What's the right way to stage these steps?

• recreate surface datasets with 5.2 data [Will]
• rimport datm, lai streams & surface datasets [Erik or @slevis-lmwg]
• update usermods to these directories [Will] ...

Maybe this can be a focus (at least for me) after LMWG?

[ekluzek]

@wwieder I'm going to move this so it gets in on b4b-dev. Which should make this quicker to happen. We can pull it in there as it is, and then we'll do other updates after CTSM5.2 comes in.

[ekluzek]

I think one thing here is that I wanted to look this over and possibly add some updates here. I think I should still look it over, but perhaps work I do in this space could be done later...

[ekluzek]

@wwieder and @TeaganKing I did add a list of checkboxes above in:

https://github.com/ESCOMP/CTSM/pull/2155#pullrequestreview-1662293010

and there a bunch of conversations to resolve. I think the way to move forward is to turn some of those into issues that will be done later, and only do the minimum needed now to bring it into b4b-dev. I've been working on dust rather than this, so haven't looked at it since those discussions. Does that plan sound right to you @wwieder?

[ekluzek]

From discussion with @TeaganKing and @adrifoster we think that first of all:

It's acceptable to bring in personal paths (to @olyson directories on Derecho) to b4b-dev. I would prefer that doesn't come to master that way, but that is also something that could be updated later.

Skimming the code we don't see any blockers for anything else (although I do want to make sure we have reminders for the future updates).

With the one exception -- that we need to be able to run a simple test case and show this works at least for one site. The current user_mods are assuming a particular update in CDEPS that will be done later, so we need a temporary workaround. This is in regard to

CLM_USRDAT.PLUMBER2:datafiles

setting which currently won't work with CDEPS now.

[ekluzek]

@wwieder @TeaganKing said you got this working in some manner? Can you post what you did to do that? Put the casedirectory here so we can see what's going on.

[wwieder]

An examples of a case I ran is now here (previously run on cheyenne) glade/campaign/cgd/tss/people/wwieder/cheyenne_archive/PLUMBER2_tests/ZM-Mon_test2

[ekluzek]

OK, I resolved conversations. Running python testing. The unit tests fail, and pylint testing fails. So I'll work on that...

[wwieder]

@ekluzek I think there are changes that need to be made to directory path (e.g. #2410).

• Do I need to make changes to this PR to reflect these new paths, or will the rimport happen after this PR is merged?
• Should we create surface datasets based on 5.2 surface data and reflect this in the usermod_dirs?
• Are there any other changes before this gets merged?

[slevis-lmwg]

@ekluzek I think there are changes that need to be made to directory path (e.g. #2410).

• Do I need to make changes to this PR to reflect these new paths, or will the rimport happen after this PR is merged?
• Should we create surface datasets based on 5.2 surface data and reflect this in the usermod_dirs?

On the first two questions...

1. I wanted to clarify that the new paths can be put in place as soon as the files get copied (or moved) to the new directory. The rimport step can happen any time after that, but it is not required for the new paths to be put in place.
2. Given how close we are to merging 5.2, that makes sense to me. I'm happy to do it. I expect I would need a quick intro first, as I did when I generated the NEON files.