Map file accessibility

[TeaganKing]

Mapping files such as this one are not publicly available: /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_blin.240826.nc

We should either 1) provide instructions or link to instructions on how to create these OR 2) make those mapfiles available. Since they're part of a broader collection of mapfiles, maybe we should discuss how to make these more accessible in general.

[mnlevy1981]

Everything in /glade/campaign/cesm/cesmdata/inputdata should be available through the inputdata repository -- I had just forgotten to push map_ne30pg3_TO_fv0.9x1.25_blin.240826.nc to the subversion repository where we keep public copies of all these files. The mapping file is now available, we just need to document how users should grab the file. @jedwards4b does CIME provide a good way to pull data down from inputdata without creating a case that uses the file? Should we tell people to run svn export by hand?

[TeaganKing]

Ah, got it! Thanks @mnlevy1981 . @YanchunHe was interested in this particular file, so I'll tag him here, too.

[jedwards4b]

CIME does not currently provide an easy way to do this, but I don't think that it would be too hard to do. https://github.com/ESMCI/cime/issues/4840

[YanchunHe]

Thanks, now I see the file, and can download it manually!

[jedwards4b]

Is there a process that looked for and failed to find that file? It might be easier to build a download capability into an existing process rather that create something completely standalone.

[TeaganKing]

We currently list the mapfiles needed for ncremap in the CUPiD config file, and then use this script to run the remapping with ncremap.

[jedwards4b]

Great, if you can provide me a working example I think that I can extend it to try to download files not available locally.

[TeaganKing]

So this script would be called via run_remapping.py, specifying a configuration file such as the key_metrics config file. Eg, something like this: python run_remapping.py config.yaml

[YanchunHe]

We currently list the mapfiles needed for ncremap in the CUPiD config file, and then use this script to run the remapping with ncremap.

Looks like this script from line 96-124 should be indent a level, otherwise it will only process the last variable.

And I tried that, it does solve my reported problem in #211

Could you please check?

[jedwards4b]

I'm confused - I ran this example out of the box and got:

 python run_remapping.py ../examples/key_metrics/config.yml 
INFO:util:
  Regridding atm time series files...
INFO:util:	 Regridding time series for case 'b.e30_beta02.BLT1850.ne30_t232.104' :
WARNING:util:[remapping]: no mapping file provided for b.e30_beta02.BLT1850.ne30_t232.104
INFO:util:	 Regridding time series for case 'b.e23_alpha17f.BLT1850.ne30_t232.092' :
WARNING:util:[remapping]: /glade/campaign/cesm/development/cross-wg/diagnostic_framework/CESM_output_for_testing/b.e23_alpha17f.BLT1850.ne30_t232.092/atm/proc/tseries/regrid/b.e23_alpha17f.BLT1850.ne30_t232.092.cam.h0a.PSL.000101-010012.nc exists and will not be overwritten
INFO:util:  ... atm time series file remapping has finished successfully.

but the mapping file listed in config.yml is /glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_blin.240826.nc which exists locally.

I then changed the name of the mapping file to something that I know does not exist: '/glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_blin.280826.nc' And ran again, only to get exactly the same message.
Why in the first case am I getting the "no mapping file provided" message and why is the second case behaving in the same way as the first? Also I don't seem to be able to redirect the output directory without copying all of the input to a new location - am I missing something?

[mnlevy1981]

The first file is saying no mapping file provided because mapping_file: [Null, '/glade/campaign/cesm/cesmdata/inputdata/cpl/gridmaps/ne30pg3/map_ne30pg3_TO_fv0.9x1.25_blin.240826.nc'] and it's triggering off of the fact that the first element of the array is translated as None in python. If you change that value to a non-existent map you should be able to trigger a file-not-found error -- the second case is seeing that the remapped file already exists so it doesn't need to check the map. Try changing vars: ['PSL'] -> vars: ['T'] in config.yml and then see if the bad mapping file triggers an error.