E3SM-Project
Current EC30to60 ocean/ice initial condition should have bumped revision number
The current ocean ice condition changed in #4091 had a change in bathymetry from ETOPO to GEBCO and should have not been placed in the EC30to60E2r2 directory of the input data folder. The revision number was bumped here https://github.com/MPAS-Dev/MPAS-Model/pull/669 but I did not propagate that to E3SM. Mixing bathymetry in the same directory leads to confusion and issues in analysis scripts.
The ocean and ice meshes date stamped 210210 should be moved to an 'r3' folder and compsets that used the EC30to60E2r2 mesh with that date should be updated to the r3 version after moving the input file.
@jonbob @xylar @milenaveneziani Here is the issue we discussed on slack. After thinking about this I'm thinking we should make the EC30to60E2r3 folder and make this as clean as possible but would appreciate your input.
@njeffery @maltrud Given the date stamp of your BGC file I wanted to make you aware of this as well as you likely have GEBCO bathymetry as well
@xylar I'm also thinking the attributes of the 210210 IC should be modified to specify the bathymetry used as done in COMPASS. Do you think I could possibly use the files_for_e3sm portion of compass to add the attributes to this? I'm guessing I could also just use ncatted as well potentially.
Also pinging @golaz for any input on the plan to move this specified initial condition to a new EC30to60E2r3 directory.
Also just confirming here that the v2 and v2.1 LR simulation campaigns used the mesh referenced here.
@xylar I'm also thinking the attributes of the 210210 IC should be modified to specify the bathymetry used as done in COMPASS. Do you think I could possibly use the files_for_e3sm portion of compass to add the attributes to this? I'm guessing I could also just use ncatted as well potentially.
I need to check if files_for_e3sm from compass can take care of this. It will be necessary to rerun files_for_e3sm regardless to produce the full set of diagnotics and analysis member files for the new mesh name.
Also just confirming here that the v2 and v2.1 LR simulation campaigns used the mesh referenced here.
Luke, just to clarify: the v2 and v2.1 LR simulation campaigns did not use the mesh and bathymetry references here, since the default initial condition file for EC30to60E2r2 has the 201001 time stamp on it.
@milenaveneziani and @vanroekel, I think the reason that the LR simulation campaigns did not use this mesh is because they are likely starting from the "spun up" initial condition, which was not updated in #4091.
yes, indeed: that was what I meant by default initial condition.
@xylar: what is the rationale in using the r# in our mesh names, can you please remind me? it refers to any revision in mesh details, including bathymetry? In that case, then creating the EC30to60E2r3 folder seems like the perfect the solution here.
Yes, the revision r# is for any change in the mesh or initial condition. So we created different revisions for different bathymetry, different vertical grid, but also we would have had a different revision for the EN4 1900 initial condition (which we never actually used).
Now I'm confused about what used which file. The very first input to the v2/v2.1 sim campaign -- https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/2781544502/20210518.v2rc1a.Frazil3.ne30pg2+EC30to60E2r2.chrysalis -- should have used the EC30to60E2r2 non spun up, well spun up very briefly, and given it has a tag after #4091 I would have guessed it used the new bathymetry. But I can check the restart files to confirm.
For v2.1: v2_1.LR.piControl is a hybrid from 20221012.submeso.piControl.ne30pg2_EC30to60E2r2.chrysalis, which in turn restarted from your summer simulation 20220715.submeso.piControl.ne30pg2_EC30to60E2r2.chrysalis. And that last simulation started from EC30to60E2r2/mpaso.EC30to60E2r2.rstFromG-anvil.201001.nc. So v2.1 for sure used the old bathymetry.
You are right. I tracked down the wrong path. I also just confirmed the v2.1 and v2 bathymetry is consistent with the old files. That makes this change simpler (no need for PRs to maint2 and maint2.1). Thanks for the explanation!
@rljacob, yes, or actually I think we should generate a new set of meshes for E3SM v3 (which will have the new bathymetry). The E2 in the mesh name explicitly indicates that it's for E3SM v2.
I agree with Xylar completely. We definitely want the bathymetry to be updated but with the E3 tag indicating a new series of meshes.
@vanroekel, what is the situation with the update initial conditions for WC14 in https://github.com/E3SM-Project/E3SM/pull/4091? Do you think that could have the same issue? If so, we should make a separate E3SM issue for that as well.
To add to the detective work, it seems that @vanroekel updated the spun-up initial conditions to the new version of the mesh in #4091 but then the spun-up initial conditions (but not the ones without spin up) were set back to the earlier version in #4127.
@xylar thanks for the continued detective work on sorting this out. I've dug around a bit as well and here is my summary of what is updated and what is not
EC30to60E2r2 -- both of the ICG files in the input data server use the ETOPO bathymetry,
mpaso.EC30to60E2r2-1900.rstFromG-anvil.200910.nc
and
mpaso.EC30to60E2r2-1900.rstFromG-anvil.201001.nc
have the same bathymetry
but the cold start ocean ICs
ocean.EC30to60E2r2.210210.nc
and
ocean.EC30to60E2r2.200908.nc
have different bathymetry (but the same horizontal mesh) This means EC30to60E2r2 G-cases run after #4091 have the different bathymetry
For the WC14
The non spun up ICs
ocean.WC14to60E2r3.200714.nc
and
ocean.WC14to60E2r3.210210.nc
use different bathymetry
there are 3 spun up conditions
mpaso.WC14to60E2r3.rstFromG-anvil.201002.nc - pre #4091
mpaso.WC14to60E2r3.rstFromG-anvil.210210.nc -- post #4091
mpaso.WC14to60E2r3.rstFromG-anvil.210226.nc -- post #4127
for these the last two use different bathymetry than the first. I'll check if the production WC14 fully coupled RRMs used post #4091, but will also make a separate issue and report that finding there.
Based on these findings I think for the EC30to60E2r2 we need to move the 210210 datestamp to an 'r3' folder and generate a new spun up condition and regenerate mask/transect files. Although I think the transect files should be okay?
If coastline hasn't changed, then the transect files are all good.
I have a question related to the Gebco bathymetry version: is there some metadata that indicates it? I wonder because we have updated the Gebco bathymetry last summer (the previous version was weird in the Chukchi Sea), but from the mentioned time stamps, it seems to me that the initial conditions that we are discussing here use the old(er) Gebco (pre-July 2022)?
Thanks for that clarification @milenaveneziani that makes sense.
Unfortunately no on the metadata. It appears I neglected to use the files_for_e3sm script in making the IC so the metadata is not there. I made the update to that file here https://github.com/MPAS-Dev/MPAS-Model/pull/669 but it did not get used for the E3SM ICs. But given the dates I would agree it is the older Gebco data.
The dates will be sufficient. Maybe we can add a note somewhere in all $meshversion/ directories that use this bathymetry, just to make sure we remember.
Pre move of the affected IC's I'm going to try run the files through the files_for_e3sm part of compass to get the right metadata in the files and we could note this bathymetry there.
Pre move of the affected IC's I'm going to try run the files through the files_for_e3sm part of compass to get the right metadata in the files and we could note this bathymetry there.
I have already done this but I can rerun (it takes about half an hour). I can add a note about the version of GEBCO and the Chukchi Sea.
I will go ahead and make the EC30to60E2r3 directory and populate it so that we can make the corresponding PR.
I believe I have all the ocean and sea ice files in place in the following locations: https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/EC30to60E2r3/ https://web.lcrc.anl.gov/public/e3sm/inputdata/ice/mpas-seaice/EC30to60E2r3/ https://web.lcrc.anl.gov/public/e3sm/diagnostics/maps/ https://web.lcrc.anl.gov/public/e3sm/diagnostics/mpas_analysis/maps/ https://web.lcrc.anl.gov/public/e3sm/diagnostics/mpas_analysis/region_masks/
I believe permissions are also correct. @vanroekel, would you be willing to make a PR to update E3SM itself accordingly? I believe we will also need @jonbob to copy some domain and mapping files.
Notes:
- I have used today's date for the ocean and sea-ice initial conditions and partition files. I think this is appropriate. They are based on the 210210 initial condition but metadata has been added so it doesn't feel appropriate to me to keep that original date. I can rerun with that date if folks disagree.
- The names of the initial conditions start with
mpasoandmpassias produced byfiles_for_e3sm. If we want these to beocean.andseaice, we should change these infiles_for_e3smso we don't have to manually rename, but I personally think the more specific names are better.
One more note: It seems that files_for_e3sm produces substantially fewer partition files than @jonbob has made in the past. I have asked on Slack for @jonbob to provide me with the logic he uses to determine this list of core counts so I can make files_for_e3sm match. I will rerun after that to get the missing partition files and add them.
The same for #5520.
Shoot, LCRC has maintenance today so we're going to have to put this work on hold until tomorrow.
Been sort of following this. What is the purpose of the new EC30to60E2r3 directory?
@rljacob, some of the files included in the current EC30to60E2r2 directory are actually for a different bathymetry (therefore a different ocean mesh by the naming conventions that we adopted for E3SMv2 and beyond). @vanroekel accidentally gave them the same name as the existing EC30to60E2r2 but they should have been a new revision, EC30to60E2r3. We are working to correct this now.
But the "E2" stands for E3SMv2 right? What v2 run would ever use this? 2.1 and 2.0 have to keep using what they've been using.