Include ILAMB in key_metrics and CESM workflow

[TeaganKing]

This PR includes running ILAMB in key_metrics and as part of the CESM workflow. The default will be having this feature turned off.

Description of changes:

• [x] Please add an explanation of what your changes do and why you'd like us to include them.

All PRs Checklist:

• [x] Have you followed the guidelines in our Contributor's Guide?
• [x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• [x] Have you made sure that the pre-commit checks passed (#8 in Adding Notebooks Guide)?
• [ ] Have you successfully tested your changes locally?

[TeaganKing]

Update on testing in cesm_alpha06d: The ILAMB html pages are generating and the correct steps are being taken in case.cupid to run ILAMB; however, there seem to be a number of errors (eg, see below) that result in the actual plots not being included in any of the original ILAMB html pages.

[DEBUG][0][WorkConfront][SurfaceUpwardLWRadiation/FLUXNET2015][b.e23_alpha17f.BLT1850.ne30_t232.092]
Traceback (most recent call last):
  File "/glade/work/tking/conda-envs/cupid-analysis/bin/ilamb-run", line 563, in WorkConfront
    c.confront(m)
  File "/glade/work/tking/conda-envs/cupid-analysis/lib/python3.11/site-packages/ILAMB/Confrontation.py", line 433, in confront
    obs, mod = self.stageData(m)
               ^^^^^^^^^^^^^^^^^
  File "/glade/work/tking/conda-envs/cupid-analysis/lib/python3.11/site-packages/ILAMB/Confrontation.py", line 341, in stageData
    obs = Variable(
          ^^^^^^^^^
  File "/glade/work/tking/conda-envs/cupid-analysis/lib/python3.11/site-packages/ILAMB/Variable.py", line 277, in __init__
    self.depth_bnds = _createBnds(self.depth)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/tking/conda-envs/cupid-analysis/lib/python3.11/site-packages/ILAMB/Variable.py", line 37, in _createBnds
    x_bnds[+1:, 0] = 0.5 * (x[:-1] + x[+1:])
                     ~~~~^~~~~~~~~~~~~~~~~~~
TypeError: can't multiply sequence by non-int of type 'float'

I'm planning to dig into the files resulting from my case and the 092 case to hopefully find some insights into why this is happening.

[TeaganKing]

When I use a different model_setup.txt file to specify runs that have been regridded and are known to work with ILAMB (b.e30_beta02.BLT1850.ne30_t232.104 and f.e23_alpha17f.FLTHIST_ne30.roughtopo.099), ILAMB runs smoothly through the case.cupid cesm workflow.

However, the link_to_ILAMB.ipynb still shows a path for ilamb_root that includes external_diag_packages; this should be updated. Hence, the plots are not displaying inline.

I would like to also note that there are a handful of additional notebook errors that I avoided by commenting out sections of the config file. I don't think these are relevant to this PR, but I recorded the error output here: /glade/work/tking/cupid_project/other_cupids/alpha6d/cime/scripts/ilamb_test_newcheckout/cupid-log-copy-from-terminal

[TeaganKing]

I wanted to note here some issues that Jazzy was running into and some key discussion points from talking with Keith, in case anyone else comes across them.

1. BLT1850 data needs to be shifted to align with other dates (such that eg year 50 of the simulations will correspond to real year 2000), as follows in model_setup.txt: b.e23_alpha17f.BLT1850.ne30_t232.092 , /glade/campaign/cesm/development/cross-wg/diagnostic_framework/CESM_output_for_testing/b.e23_alpha17f.BLT1850.ne30_t232.092/lnd/hist/regrid/, 50, 2000 b.e30_beta02.BLT1850.ne30_t232.104 , /glade/campaign/cesm/development/cross-wg/diagnostic_framework/CESM_output_for_testing/b.e30_beta02.BLT1850.ne30_t232.104/lnd/hist/regrid/, 50, 2000

2. b.e23_alpha17f.BLT1850.ne30_t232.092 needs to be regridded & point to regrid dir in model_setup.txt

3. Photosynthesis from an SP run is represented by CLM history variable FPSN, while for a BGC run it is represented by GPP.

[TeaganKing]

This does not work out of the box as is seen here with the examples, but it DOES work with the added , 50, 2000 on both lines of model_setup.txt. So, perhaps we should include an 'if BLT1850 is in casename &/or basecasename, add , 50, 2000 on both lines of model_setup.txt' to the generate_ilamb_config_files.py script. I'm also comparing b.e23_alpha17f.BLT1850.ne30_t232.092 (instead of the roughtopo 99 case) & b.e30_beta02.BLT1850.ne30_t232.104. It may be worth checking that there isn't any harm in changing this as the default for all other notebooks. We should probably update that anyways...

[TeaganKing]

Next step: test running ILAMB in CESM workflow

[TeaganKing]

Items from testing in CESM workflow:

• [x] should allow ./xmlchange CUPID_RUN_TYPE=BGC as well as ./xmlchange CUPID_RUN_TYPE="BGC"
• [x] CUPID_RUN_TYPE defaults to None, which may cause issues?
• [x] In the CESM workflow, generate ilamb config file script requires: ilamb_config_loc = os.path.join(cupid_config_loc, "../../../../tools/CUPiD/ilamb_aux"). This could also use os.path.join(cesm_root, "tools", "CUPiD") if we put cesm_root back in as an argument.
• [x] change defaults_file to be under external_tool in config & in generate_ldf_config

[TeaganKing]

I think that these tests worked once I removed all of the CUPID_NTASKS related code, but there were a few points of pausing, removing other CUPID_NTASKS items, and continuing. We may want to double check this PR in a cesm3_alpha07c sandbox just to be sure it runs smoothly.

[TeaganKing]

There was a brief consideration of changing this to CUPID_RUN_ILAMB_BGC and CUPID_RUN_ILAMB_SP so as not to have users set both CUPID_RUN_TYPE and CUPID_RUN_ILAMB; however, after discussing with Keith, it seems that a default of BGC will suffice, and users can set CUPID_RUN_TYPE to SP if they are using this less-common configuration. This also keeps the user interface similar to the interface for other external tools such as ADF and LDF. For the very unusual comparisons of BGC & SP cases to one another, BGC can be used, and just a few outputs will be left blank.

[TeaganKing]

Final tests with ILAMB are currently being run with a wallclock of 6hr, updated cime to cime6.1.118 with cesm_3_0_alpha07c, and CUPID_MEM_PER_TASK=35:

• [x] 1) using BGC run type:
• [x] 2) using SP run type: ILAMB output is properly linked, but key plots are not showing up

Another note:

• [x] We should also include instructions for running ILAMB via the CESM workflow in our documentation.

[TeaganKing]

This is now working when files are available in the regrid directory. Note that additional time & memory resources are required for running ILAMB. I was able to run this via the CESM workflow with CUPID_RUN_ILAMB set to True and successfully linked to ILAMB output.

[YanchunHe]

This is now working when files are available in the regrid directory. Note that additional time & memory resources are required for running ILAMB. I was able to run this via the CESM workflow with CUPID_RUN_ILAMB set to True and successfully linked to ILAMB output.

This is cool! I tried out earlier, but has difficulty with iLAMB to ingest the regridded NorESM (CLM/CTSM) output. I will try it again later.