NCAR
Unify NOAA & NCAR versions & CCPP-compliant interfaces of RRTMGP
Description
Version Unification
- NOAA is at (or soon will be at) v1.8
- CAM is at v1.7
- RRTMGP latest: 1.9.1
- NOAA is seeing a performance hit with this version, per @dustinswales
- NCAR ran into this intel compiler issue: https://github.com/earth-system-radiation/rte-rrtmgp/issues/369
Interface unification My current plan is as follows:
- [ ] Break up existing rrtmgp_lw_main.F90 and rrtmgp_sw_main.F90 into smaller components/schemes (I am working on this now; have completed the longwave schemes needed for CAM)
- [ ] Validate the schemes (and interstitials) in CAM-SIMA
- [ ] Bring the broken-up schemes back into ccpp-physics and add additional interstitials to get the SCM working
The following schemes will be shared between NOAA and NCAR (list subject to change):
- rrtmgp_lw_gas_optics_data
- rrtmgp_sw_gas_optics_data
- rrtmgp_lw_gas_optics
- rrtmgp_sw_gas_optics
- rrtmgp_lw_rte (will basically just be the core calls to rte_lw)
- rrtmgp_sw_rte (will basically just be the core calls to rte_sw)
EMC has been in discussions with Dustin about updating RRTMGP to a newer version in CCPP and including a single precision option. We are planning to start using RRTMGP for NCEP UFS applications within the next couple of years. It’s great that we can join efforts on this. We may also need to further reduce the number of gas optics bands to improve efficiency for operational NWP forecasts. @Qingfu-Liu @dustinswales
@peverwhee Thanks for opening this issue. This is a great opportunity to develop CCPP schemes for use across hosts.
I should mention that the main GP driver(s) that you are partitioning was refactored from smaller components. This yielded a modest performance boost, but diverged from our original vision of SDF modularity for the optical components:
That's just to say that you could revive the (already partitioned) scheme and metadata files that were deleted in the RRTMGP refactoring PR, which might save you some work.
For the common schemes you propose I have some questions that we will need to figure out (none of these are hard to handle IMO):
- The data and initialization routines are currently in the same module. Is there a need in CAM-SIMA to have these separated?
- rrtmgp_lw_gas_optics_data
- rrtmgp_sw_gas_optics_data
- rrtmgp_lw_gas_optics
- rrtmgp_sw_gas_optics ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of gas_optics().
- We will need to maintain our current driver and add the "rte only" interface you propose:
- rrtmgp_lw_main (~ rrtmgp_lw_gas_optics->rrtmgp_lw_cloud_optics->rrtmgp_lw_cloud_sampling->rrtmgp_lw_rte)
- rrtmgp_sw_main (~ rrtmgp_sw_gas_optics->rrtmgp_sw_cloud_optics->rrtmgp_sw_cloud_sampling->rrtmgp_sw_rte)
- rrtmgp_lw_rte
- rrtmgp_sw_rte ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of rte().
- What about cloud optics? Currently, we are using the rte-rrtmgp extensions for cloud-optics, we've validated that these are consistent with the previous treatment of clouds in RRTMG, but we may at some point expand our handling of cloud-optics. So we have CCPP interfaces to the rte-rrtmgp cloud optics (analogous to the gas-optics infrastructure)
- rrtmgp_lw_cloud_optics
- rrtmgp_sw_cloud_optics ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of cloud_optics().
- And cloud sampling (i.e McICA)? We had a CCPP scheme to do just this piece, but it was folded into our main driver. This scheme was adopted from the rte-rrtmgp extension and has all of the popular cloud-overlap assumptions included.
- rrtmgp_lw_cloud_sampling
- rrtmgp_sw_cloud_sampling ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of cloud_sampling().
@yangfanglin Thanks for your comments. Our effort to update rte-rrtmgp in the CCPP aligns well with Courtney's work to implement rte-rrtmgp in the CCPP for CAM-SIMA. We should be able to use the same rte-rrtmgp submodule hash and some common CCPP interstitial schemes across hosts. Of course many pieces are host specific, which we will not touch, but things like data initialization are easily sharable across CCPP enabled hosts. This could be a good topic for an upcoming CCPP Physics Mgmt. Mtg.?
@peverwhee Thanks for opening this issue. This is a great opportunity to develop CCPP schemes for use across hosts.
I should mention that the main GP driver(s) that you are partitioning was refactored from smaller components. This yielded a modest performance boost, but diverged from our original vision of SDF modularity for the optical components.
That's just to say that you could revive the (already partitioned) scheme and metadata files that were deleted in the RRTMGP refactoring PR, which might save you some work.
Thanks! I'll take a look at these!
For the common schemes you propose I have some questions that we will need to figure out (none of these are hard to handle IMO):
- The data and initialization routines are currently in the same module. Is there a need in CAM-SIMA to have these separated?
- rrtmgp_lw_gas_optics_data
- rrtmgp_sw_gas_optics_data
- rrtmgp_lw_gas_optics
- rrtmgp_sw_gas_optics ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of gas_optics().
Yes. Well, kind of. In CAM, our cloud optics init phase has to come after our gas optics init phase, but the reverse is true for the run phases. That said, now that I write it down, I can just split up the cloud optics modules since we're not sharing those anyway.
- We will need to maintain our current driver and add the "rte only" interface you propose:
- rrtmgp_lw_main (~ rrtmgp_lw_gas_optics->rrtmgp_lw_cloud_optics->rrtmgp_lw_cloud_sampling->rrtmgp_lw_rte)
- rrtmgp_sw_main (~ rrtmgp_sw_gas_optics->rrtmgp_sw_cloud_optics->rrtmgp_sw_cloud_sampling->rrtmgp_sw_rte)
- rrtmgp_lw_rte
- rrtmgp_sw_rte ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of rte().
OK - that's good information. Thank you!
- What about cloud optics? Currently, we are using the rte-rrtmgp extensions for cloud-optics, we've validated that these are consistent with the previous treatment of clouds in RRTMG, but we may at some point expand our handling of cloud-optics. So we have CCPP interfaces to the rte-rrtmgp cloud optics (analogous to the gas-optics infrastructure)
- rrtmgp_lw_cloud_optics
- rrtmgp_sw_cloud_optics ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of cloud_optics().
CAM uses a kind of a bespoke cloud optics method (rather than using the inherent RRTMGP methods. We will have our own cloud_optics modules. The current ccpp-physics schemes should be able to remain unmodified. We will keep our CAM-specific schemes in a separate repository.
- And cloud sampling (i.e McICA)? We had a CCPP scheme to do just this piece, but it was folded into our main driver. This scheme was adopted from the rte-rrtmgp extension and has all of the popular cloud-overlap assumptions included.
- rrtmgp_lw_cloud_sampling
- rrtmgp_sw_cloud_sampling ^Metadata exists in non-refactored RRTMGP CCPP code for inline (SDF) use of cloud_sampling().
The McICA sampling is also done differently in CAM to the current ccpp-physics implementation. So we will have our own cloud_sampling modules. The current ccpp-physics schemes can remain unmodified.
@peverwhee Thanks for your explanations. This all seems sensible to me.