Break out module_dm external subroutines into separate files
TYPE: enhancement
KEYWORDS: intel, compilation, llvm, memory
SOURCE: internal
DESCRIPTION OF CHANGES: Problem: The Intel oneAPI compilers (and others like nvhpc) struggle with some of the larger (15k+ lines of code) files within WRF. This causes intense memory usage that is not often available to the average user not in a resource-rich environment. This often limits compilation to single threaded if even possible or to a dedicated environment with enough memory if available. If neither of those is available to a user, they will be unable to use these configurations entirely.
Solution:
This PR focuses on the module_dm sections of code to reduce its individual file size to manageable levels. This and its helper subroutines are instead broken out into many smaller files.
TESTS CONDUCTED: Attached to this PR are plots of the respective effects of theses changes. Changes were tested with intel and gcc compilers, but only intel memory usage is shown as it exacerbates the memory usage issue.
Highlighted is the region during compilation which memory usage spikes that this PR addresses (module_dm) before these changes take place :
This usage is then dropped when using this PR's edits, splitting across multiple other files :
Zooming in we can now see the external subroutines originally in module_dm comprise a number of smaller compilation units :
The regression test results:
Test Type | Expected | Received | Failed
= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Number of Tests : 23 24
Number of Builds : 60 57
Number of Simulations : 158 150 0
Number of Comparisons : 95 86 0
Failed Simulations are:
None
Which comparisons are not bit-for-bit:
None
I tested code before and after this PR, and model produces identical results in my test. Also with 4 processors, the compile time is about 12 minutes!
After cleaning the build, it looks like the following processed source files aren't being removed:
external/RSL_LITE/feedback_domain_em_part1.f90
external/RSL_LITE/feedback_domain_em_part2.f90
external/RSL_LITE/force_domain_em_part2.f90
external/RSL_LITE/interp_domain_em_part1.f90
external/RSL_LITE/interp_domain_em_part2.f90
external/RSL_LITE/interp_domain_em_part3.f90
external/RSL_LITE/interp_domain_em_small.f90
I removed the *.f90 clean addition in the external/RSL_LITE/makefile, especially as I don't know if on case insensitive systems it would remove actual source files like external/RSL_LITE/f_pack.F90. I think adding back something like *domain_em*.f90 would suffice but look odd inside the clean command