CTSM
CTSM copied to clipboard
ESCOMP
ctsm5.2.mksurfdata
Description of changes
This pull request will document final steps in the ctsm5.2 branch development before we make the ctsm5.2.0 tag.
Specific notes
Contributors other than yourself, if any: @mvertens @ekluzek @jedwards4b @billsacks @wwieder @lawrencepj1 @negin513 @dlawrenncar @olyson Please add contributors that I may have forgotten and reorder if I have misrepresented your contributions.
CTSM Issues Fixed (include github issue #): Fixes #1903 Fixes #1716 Fixes #1734
Pull Requests that comprise this effort: https://github.com/ESCOMP/CTSM/pulls?q=is%3Apr+base%3Actsm5.2.mksurfdata+is%3Aclosed #2327 #2318 #2164 #2048 #2016 #2008 #1946 #1920 #1890 #1873 #1866 #1853 #1796 #1756 #1748 #1746 #1732 #1728 #1721 #1663 #1586
Are answers expected to change (and if so in what way)? Yes, more than roundoff for at least the following reasons:
- Would have expected roundoff if we had just updated mksurfdata_map with mksurfdata_esmf
- More than roundoff because we have replaced several raw datasets to include more modern data
Any User Interface Changes (namelist or namelist defaults changes)?
- mksurfdata_esmf has a less cumbersome workflow than mksurfdata_map
- namelist_defaults point to new fsurdat, landuse, and finidat files
Testing performed, if any:
- build-namelist_test.pl (derecho)
- Python unit and sys tests (derecho)
- aux_clm (izumi, derecho)
- ctsm_sci (derecho)
most recently for tag
alpha-ctsm5.2.mksrf.20_ctsm5.1.dev163as documented in #2318 for example here.
- Ran ./manage_externals/checkout_externals
- Updated /ccs_config/component_grids_nuopc.xml as follows to avoid test failures:
index cc0bb19..d6f32d0 100644
--- a/component_grids_nuopc.xml
+++ b/component_grids_nuopc.xml
@@ -274,7 +274,7 @@
<mesh>$DIN_LOC_ROOT/share/meshes/mpasa15_ESMFmesh-20210803.nc</mesh>
<desc>MPAS-A 15-km quasi-uniform mesh:</desc>
</domain>
- <domain name="mpasa15-3conus">
+ <domain name="mpasa15-3">
<nx>6488066</nx> <ny>1</ny>
<mesh driver="nuopc">$DIN_LOC_ROOT/share/meshes/mpasa15-3.conus_ESMF_desc.210816.nc</mesh>
<desc>MPAS-A 15-3-km variable-uniform mesh centered over CONUS:</desc>
- In tools/mksurfdata_esmf/Makefile (do this on /scratch to avoid running out of space), temporarily add
global-present-ultra-hi-resto the "all" target and run:
conda activate ctsm_pylib
rm -rf tool_bld # to ensure I started clean
./gen_mksurfdata_build.sh # build mksurfdata_esmf
module load nco # needed by the Makefile
make all
CTSM5.2 dataset creation is here: /glade/derecho/scratch/slevis/temp_work/ctsm5.2.mksurfdata/tools/mksurfdata_esmf
I hope this is it, and I'm not holding my breath :-)
>> git tag -a alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168
>> git push escomp alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168
>> git describe
alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168
CTSM5.2 dataset creation DONE, now confirming that everything worked: /glade/derecho/scratch/slevis/temp_work/ctsm5.2.mksurfdata/tools/mksurfdata_esmf
Before updating namelist_defaults_ctsm.xml, for my own sanity, I ran these tests to confirm that they still work and got:
- Python unit and sys tests (derecho) PASS
- build-namelist_test.pl (derecho)
2587/2587 < PASS> <Test Id: 2587> <Desc: -res 0.9x1.25 -mask gx1v7 -sim_year 1850 -envxml_dir . -lnd_tuning_mod clm5_1_cam6.0 -bgc bgc -phys clm5_1: user_nl_ctsm_real_parameters file exists>
Successfully ran all testing for build-namelist
Cleanup files created
/bin/rm: cannot remove 'env_run.xml': No such file or directory
/bin/rm: cannot remove 'user_nl_ctsm_real_parameters': No such file or directory
/bin/rm: cannot remove 'temp_file.txt': No such file or directory
@ekluzek @wwieder I assume that I do not need to create a new version of the NEON files again, since they will be the same (other than using the "official" fsurdat files in getting generated). But if that's what we want, please let me know.
There are NEON surface datafiles that were created with a 5.2 dataset already, right? If so, no I don't think we need to create them again, unless you need creation dates for inputdata that are consistent with the default 5.2 surface datasets?
The reason to create the NEON files again, would be so that we can clearly document what tag of CTSM was used to make the datasets. I think this is critically important for the global resolutions, but as NEON changes more often I don't think it's as important for them.
When I look at the ncdump I see this for the NEON file...
ncdump -h $CSMDATA/lnd/clm2/surfdata_esmf/NEON/surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206.nc
netcdf surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206 {
dimensions:
time = UNLIMITED ; // (12 currently)
lsmlon = 1 ;
lsmlat = 1 ;
nlevsoi = 10 ;
numurbl = 3 ;
numrad = 2 ;
nlevurb = 10 ;
nglcecp1 = 11 ;
nglcec = 10 ;
cft = 64 ;
natpft = 15 ;
lsmpft = 79 ;
variables:
double lsmlon(lsmlon) ;
lsmlon:_FillValue = nan ;
double lsmlat(lsmlat) ;
lsmlat:_FillValue = nan ;
double LONGXY(lsmlat, lsmlon) ;
LONGXY:_FillValue = nan ;
LONGXY:long_name = "longitude" ;
LONGXY:units = "degrees east" ;
double LATIXY(lsmlat, lsmlon) ;
LATIXY:_FillValue = nan ;
LATIXY:long_name = "latitude" ;
LATIXY:units = "degrees north" ;
int mxsoil_color(lsmlat, lsmlon) ;
mxsoil_color:long_name = "maximum numbers of soil colors" ;
mxsoil_color:units = "unitless" ;
int SOIL_COLOR(lsmlat, lsmlon) ;
SOIL_COLOR:long_name = "soil color" ;
SOIL_COLOR:units = "unitless" ;
float PCT_SAND(nlevsoi, lsmlat, lsmlon) ;
PCT_SAND:_FillValue = nanf ;
PCT_SAND:long_name = "percent sand" ;
PCT_SAND:units = "unitless" ;
.
.
.
double MONTHLY_HEIGHT_BOT(time, lsmpft, lsmlat, lsmlon) ;
MONTHLY_HEIGHT_BOT:_FillValue = nan ;
MONTHLY_HEIGHT_BOT:long_name = "monthly height bottom" ;
MONTHLY_HEIGHT_BOT:units = "meters" ;
int time(time) ;
time:long_name = "Calendar month" ;
time:units = "month" ;
double lon(lsmlat, lsmlon) ;
lon:_FillValue = nan ;
double lat(lsmlat, lsmlon) ;
lat:_FillValue = nan ;
// global attributes:
:Conventions = "NCAR-CESM" ;
:Source = "Community Land Model: CLM5" ;
:Number-of-tasks = 512 ;
:Input_grid_dataset = "fv0.9x1.25_141008_polemod_ESMFmesh.nc" ;
.
.
.
:VOC_EF_raw_data_file_name = "mksrf_vocef_0.5x0.5_simyr2000.c110531.nc" ;
:Created_on = "2024-02-06" ;
:Created_by = "slevis" ;
:Created_with = "./subset_data -- e196a1efd" ;
:Created_from = "/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_2000_78pfts_c240130.nc" ;
:Updated_on = "2024-02-06" ;
:Updated_by = "slevis" ;
:Updated_with = "/glade/work/slevis/git/mksurfdata_toolchain/python/ctsm/site_and_regional/modify_singlept_site_neon.py" ;
:Updated_from = "subset_data_single_point/surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206.nc" ;
:Updated_using = "/glade/work/slevis/git/mksurfdata_toolchain/neon_surffiles/ABBY_surfaceData.csv" ;
:Updated_fields = "PCT_CLAY, PCT_SAND, ORGANIC" ;
}
So the only clue I have to the version used is the hash: e196a1efd. Which would take some time to figure out what version that corresponds to. But, it's also not going to be obvious to everyone that that hash is significant nor how you use it.
Whereas the latest global files have this clear with:
// global attributes:
:Conventions = "NCAR-CESM" ;
:History_Log = "created on: 02-16-24 20:15:42" ;
:Source = "Community Land Model: CLM5" ;
:Version = "alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168" ;
:Logname = "slevis" ;
:Host = "derecho6" ;
:Number-of-tasks = 512 ;
So you know to use alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168 to recreate the file.
I'm OK with this not re-creating these files if you are too, Erik.
On Tue, Feb 20, 2024 at 1:46 PM Erik Kluzek @.***> wrote:
The reason to create the NEON files again, would be so that we can clearly document what tag of CTSM was used to make the datasets. I think this is critically important for the global resolutions, but as NEON changes more often I don't think it's as important for them.
When I look at the ncdump I see this for the NEON file...
ncdump -h $CSMDATA/lnd/clm2/surfdata_esmf/NEON/surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206.nc netcdf surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206 { dimensions: time = UNLIMITED ; // (12 currently) lsmlon = 1 ; lsmlat = 1 ; nlevsoi = 10 ; numurbl = 3 ; numrad = 2 ; nlevurb = 10 ; nglcecp1 = 11 ; nglcec = 10 ; cft = 64 ; natpft = 15 ; lsmpft = 79 ; variables: double lsmlon(lsmlon) ; lsmlon:_FillValue = nan ; double lsmlat(lsmlat) ; lsmlat:_FillValue = nan ; double LONGXY(lsmlat, lsmlon) ; LONGXY:_FillValue = nan ; LONGXY:long_name = "longitude" ; LONGXY:units = "degrees east" ; double LATIXY(lsmlat, lsmlon) ; LATIXY:_FillValue = nan ; LATIXY:long_name = "latitude" ; LATIXY:units = "degrees north" ; int mxsoil_color(lsmlat, lsmlon) ; mxsoil_color:long_name = "maximum numbers of soil colors" ; mxsoil_color:units = "unitless" ; int SOIL_COLOR(lsmlat, lsmlon) ; SOIL_COLOR:long_name = "soil color" ; SOIL_COLOR:units = "unitless" ; float PCT_SAND(nlevsoi, lsmlat, lsmlon) ; PCT_SAND:_FillValue = nanf ; PCT_SAND:long_name = "percent sand" ; PCT_SAND:units = "unitless" ; . . . double MONTHLY_HEIGHT_BOT(time, lsmpft, lsmlat, lsmlon) ; MONTHLY_HEIGHT_BOT:_FillValue = nan ; MONTHLY_HEIGHT_BOT:long_name = "monthly height bottom" ; MONTHLY_HEIGHT_BOT:units = "meters" ; int time(time) ; time:long_name = "Calendar month" ; time:units = "month" ; double lon(lsmlat, lsmlon) ; lon:_FillValue = nan ; double lat(lsmlat, lsmlon) ; lat:_FillValue = nan ;
// global attributes: :Conventions = "NCAR-CESM" ; :Source = "Community Land Model: CLM5" ; :Number-of-tasks = 512 ; :Input_grid_dataset = "fv0.9x1.25_141008_polemod_ESMFmesh.nc" ; . . . :VOC_EF_raw_data_file_name = "mksrf_vocef_0.5x0.5_simyr2000.c110531.nc" ; :Created_on = "2024-02-06" ; :Created_by = "slevis" ; :Created_with = "./subset_data -- e196a1efd" ; :Created_from = "/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_2000_78pfts_c240130.nc" ; :Updated_on = "2024-02-06" ; :Updated_by = "slevis" ; :Updated_with = "/glade/work/slevis/git/mksurfdata_toolchain/python/ctsm/site_and_regional/modify_singlept_site_neon.py" ; :Updated_from = "subset_data_single_point/surfdata_1x1_NEON_ABBY_hist_2000_78pfts_c240206.nc" ; :Updated_using = "/glade/work/slevis/git/mksurfdata_toolchain/neon_surffiles/ABBY_surfaceData.csv" ; :Updated_fields = "PCT_CLAY, PCT_SAND, ORGANIC" ; }
So the only clue I have to the version used is the hash: e196a1e https://github.com/ESCOMP/CTSM/commit/e196a1efd56dd29126a2ef179174f42a1be03f55. Which would take some time to figure out what version that corresponds to. But, it's also not going to be obvious to everyone that that hash is significant nor how you use it.
Whereas the latest global files have this clear with:
// global attributes: :Conventions = "NCAR-CESM" ; :History_Log = "created on: 02-16-24 20:15:42" ; :Source = "Community Land Model: CLM5" ; :Version = "alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168" ; :Logname = "slevis" ; :Host = "derecho6" ; :Number-of-tasks = 512 ;
So you know to use alpha-ctsm5.2.mksrf.21_ctsm5.1.dev168 to recreate the file.
— Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/pull/2372#issuecomment-1955051536, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5IWJBVRU4S4URKSOBP553YUUDQTAVCNFSM6AAAAABDLHMNT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGA2TCNJTGY . You are receiving this because you were mentioned.Message ID: @.***>
Yeah I think it's OK. It looks like files were recreated a lot (six times) back in 2021, and haven't been recreated since then. But, I see recreating the NEON files as more likely than the global files. The current ctsm5.1 NEON files also only have hashes to identify them as I describe above, so this is also a problem with the current set as well. But, the provenance of these files doesn't need as careful of accounting as we do the global ones. So it's OK to wait for a reason to update them to get the CTSM tag used to create them in the global attributes. The most important provenance for the NEON files is which global dataset was used to create it, and that is clearly documented. With that and the NEON csv file you can replicate the files easily. So yes, let's not worry about it for the NEON files.
After updating namelist_defaults_ctsm.xml, I ran these tests to confirm that they still work:
- Python unit and sys tests (derecho) PASS
- build-namelist_test.pl (derecho) PASS
- Now starting the test-suites, which will give me the final list of tests for which I need to provide new finidat files:
izumi OK ./cs.status.fails |grep -v '168: DIF'|grep -v NLCOM|grep -v 'EXPECTED FAILURE'|grep -v 'does not exist'
derecho aux_clm OK
derecho ctsm_sci IN PROGRESS (see post below)
Comment out mpasa15-3* from anything ctsm and restore /ccs_config as per #2386 .
I would like to request reviews from various people. This PR is huge, so I will specify areas of focus for different reviewers. I'm asking now rather than at the last minute in case this helps people find the time. I am also happy to meet and review together with people who prefer that approach.
Erik and I came up with this list: Sam, Erik, and all of the PR's contributors if they wish -- general review @wwieder -- soil texture @lawrencepj1 -- soil color, pft, lai rawdata DONE @olyson -- urban rawdata DONE @Katetc -- glaciers
CTSM SE volunteers to review the mksurfdata_esmf/README:
- Sophia Macarewich followed the README to generate a "noanthro" fsurdat successfully.
I appreciate the invitation to review this, but I feel like I reviewed the individual pieces sufficiently when they first came in, so don't have a need to review it further and I don't think I'll have time to get my head into this enough to give it another review at this point. If there are specific questions that you'd like to run by me, I'd be happy to try to help with them.
I'm down to one ctsm_sci test that I need to resolve on derecho:
SSP_Ld4.f09_g17.I1850Clm50BgcCrop.derecho_intel.clm-ciso_rtmColdSSP
- first failed asking for
user_init_interp = .true. - pointing to a newly created finidat for a different case, gives this error:
ERROR: Cannot initialize from a run without c13/c14 to a run with c13/c14...so I need to regenerate the finidat using this simulation. - the file generated here worked:
.../scratch/slevis/SSP_Ld4.f09_g17.I1850Clm50BgcCrop.derecho_intel.clm-ciso_rtmColdSSP.GC.20240223_171612_7h08su.ref1/run/init_generated_files/ - copied the file to /inputdata (saving a copy of the previous before overwriting) and reran the full test
- reran one of the tests without c13/c14 that now point to this updated finidat and confirmed bit-for-bit same answers
I just now did:
git tag -a alpha-ctsm5.2.mksrf.22_ctsm5.1.dev168
git push escomp alpha-ctsm5.2.mksrf.22_ctsm5.1.dev168
git describe
and got alpha-ctsm5.2.mksrf.22_ctsm5.1.dev168
OK, there are two last things left. We want to get some feedback on the finished product from several people (that @slevis-lmwg already reached out to). And the second thing is #2378, which is the last planned tag along the branch. The plan then is to bring this PR to master and make the ctsm5.2.0 tag with it. We think this is on the order of a week, taking into account that LMWG is the next three days.
I am running the ./rimport commands now. The first is in progress. The others are done:
./rimport -list ctsm5.2.0_list.txt
./rimport -list ctsm5.2.0_neon.txt
./rimport -list ctsm5.2.0_finidat.txt
./rimport -file lnd/clm2/rawdata/pftcftdynharv.0.25x0.25.LUH2.noanthro.c20230226/mksrf_landuse_ctsm52_noanthroLUH2_1.c20230226.nc
I updated the /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/README to reflect this information.
@ekluzek confirmed that I should go ahead and ./rimport the following datasets to svn because they are used in ctsm testing:
pftcftdynharv.0.25x0.25.SSP1-2.6.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP2-4.5.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP3-7.0.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP4-3.4.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP5-8.5.simyr2015-2100.c20230226
and that I could skip the following until further notice:
pftcftdynharv.0.25x0.25.SSP1-1.9.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP4-6.0.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP5-3.4.simyr2015-2100.c20230226
pftcftdynharv.0.25x0.25.SSP1-2.6.simyr2100-2300.c20230226
pftcftdynharv.0.25x0.25.SSP5-3.4.simyr2100-2300.c20230226
pftcftdynharv.0.25x0.25.SSP5-8.5.simyr2100-2300.c20230226
pftcftdynharv.0.25x0.25.LUH2.histsimyr0850-1850.c20230226
1850-2015 was done a while ago.
Above ./rimports are DONE
TODO slevis: double check that I have rimported the new raw data files for urban, glacier, soil texture, lakes, ...
- Urban, glacier, soil texture OK
- Lakes to 2100, ./rimport DONE
Updated to dev171 and made new tag:
git tag -a alpha-ctsm5.2.mksrf.23_ctsm5.1.dev171
git push escomp alpha-ctsm5.2.mksrf.23_ctsm5.1.dev171
I made a new tag because the last commit changes answers in fsurdat files generated with the --potveg_flag:
git tag -a alpha-ctsm5.2.mksrf.24_ctsm5.1.dev171
git push escomp alpha-ctsm5.2.mksrf.24_ctsm5.1.dev171
I made a replacement potveg file for our default datasets, but
- [x] I think I need to remake it so that the file may include the new tag in its metadata
- [x] ./rimport the new file
@ekluzek @lawrencepj1 and I reviewed the pft, lai, and soil color references in the files that I generated for all SSPs, as well as for 1850, 2000, and potveg. We did not spot problems.
Bill Lipscomb and the CISM team gave their thumbs up on the Glacier datasets.
Here's what Bill had to say after being reminded about these datasets:
Hi Erik,
I'd forgotten that Adam and René redid these data sets a couple of years ago. They were using the latest RGI glacier data, which fixed some problems in the Himalayas, and also the latest BedMachine data for ice sheets. I think they were very careful and thorough. So I don't think we'll need an update now or in the near future.
Thanks, Bill
This completes the review portion on the input and output datasets.
Hi @olyson. In your review you asked for changes, so I just need you to approve the changes (under the 1 change requested at the bottom). Or you could do a quick review to just approve it. I don't need anything extensive here, but right now merging is blocked because of the request for changes. I'm pretty sure @slevis-lmwg took care of all the things you asked for. But, it's also not bad to double check...
I submitted these izumi tests to help get around the glitches that Erik saw on izumi. I may have misnamed the new baselines. I added _slevis suffixes to them rather than deleting Erik's:
./run_sys_tests -s fates -c alpha-ctsm5.2.mksrf.27_ctsm5.1.dev176 -g fates-sci.1.72.2_api.34.0.0-alpha-ctsm5.2.mksrf.27_ctsm5.1.dev176_slevis
./run_sys_tests -s mosart -c alpha-ctsm5.2.mksrf.27_ctsm5.1.dev176 -g mosart1_0_49-alpha-ctsm5.2.mksrf.27_ctsm5.1.dev176_slevis
I compared to alpha-ctsm5.2.mksrf.27_ctsm5.1.dev176 ("null" test) for now. We can run against other baselines later if we have specific questions or concerns.
Failure not labeled EXPECTED:
ERS_D_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdHydro
but a similar test in the baseline also fails and is labeled EXPECTED:
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_intel.clm-FatesColdHydro
and is documented here:
https://github.com/ESCOMP/CTSM/issues/2373
https://github.com/NGEET/fates/issues/1163
https://github.com/ESCOMP/CTSM/pull/2355#issuecomment-1960451417