Notify UFS Application POCs of Regression Test Baseline Changes due to Code Changes in Submodules

[grantfirl]

Description

@lisa-bengtsson mentioned that if UFS applications are using the latest development branches of submodules (e.g. ccpp-physics) that UFS application POCs must pay acute attention to all UFS PRs in order to be aware of baseline changes to UFS RTs being introduced. Given the volume and wide range of changes coming in to the top-level ufs-weather-model repository, it is often impractical for all UFS application leads to do so, which can sometime lead to "surprise" changes in the expected answers. Although code managers are typically acutely aware of how PRs in their domain affect UFS RT baselines, this information doesn't always trickle upward to the application level.

Solution

One solution would be to add "CODEOWNER" functionality to some (all?) tests in the UFS RT suite (rt.conf). For example, one or more people are listed as contacts when a baseline for a given RT changes. This would give application POCs a better opportunity to avoid "surprise" baseline changes for their application. Perhaps this could be integrated into @BrianCurtis-NOAA excellent auto_rt script.

Alternatives

Insist that UFS application POCs/leads pay close attention to all UFS PRs and whether their application's UFS RTs are affected by baseline changes.

[junwang-noaa]

Thanks for bringing this up. There were some discussions on the RT ownership. Some the the responsibilities of RT owners are summarized here:

• Decide what tests to be included in for the application

a. Configurations for parallel/pre-implementation/prototype test b. Feature tests c. Resolution tests

• Maintain and update the regression tests above (including related fix files and ICs)
• Clean up tests with obsolete configurations
• Coordinate with UFS code managers on how frequent the tests need to run, every PR, every week, etc
• Coordinate with the UFS code manager, other application owners and developers to resolve issues with failed regression test
• Review the regression test suite with application team and UFS code managers on a regular basis

[junwang-noaa]

@ChunxiZhang-NOAA @JessicaMeixner-NOAA @JacobCarley-NOAA @BinLiu-NOAA @ShastriPaturi @JianpingHuang-NOAA @lipan-NOAA FYI.

[lisa-bengtsson]

Thank you for your comments Jun. At the moment, the individual developer can just indicate that new baselines are needed, even for a control test that is a pre-implementation test (e.g. prototype 8). It seems that it would be beneficial to have the RT owner of the "control RT's" (e.g. cpld_control_p8) approve if a baseline change is allowed for this test. Most of the time science tests should have been carried out before reaching the RT's, but we've had updates for RRFS that made it into Prototype 8 without science verification for the global simulations.

[lisa-bengtsson]

@yangfanglin