WRF
WRF copied to clipboard
wrf-model
Hydro CMake and Updates
TYPE: new feature, bug fixes, enhancement
KEYWORDS: CMake, Build System, Hydro
SOURCE: Soren Rasmussen and Ryan Cabell, NCAR
DESCRIPTION OF CHANGES:
-
Build Systems:
- CMake build system added and renaming .F and .f90 files to .F90 so they automatically get preprocessed and the editor recognizes them as free-form
- added compiler info to all netcdf output files
- Alma support added
- Cray compiler fixes:
- Removes comma after write statement and changes A*-BC statements to A(-B)*C
- Use assumed shape / volatile arrays with ext libs - Replace timestep multiple check with mod operator
- Update CMake with proper MPI and PGI/NVHPC support
-
General:
- whitespace cleanup
-
MPI
- Optimized inefficient MPI routines. Converted MPP_LAND p2p MPI to collectives, Remove unneeded mpp_land_sync() calls
-
Routing changes
- Add scrape of outer edge of domain so water doesn't pile up
- Remove 'FATAL ERROR' from SMCRT depletion message
- Add to retro IO option (NWM)
- routing bugfix: removed CHANN_K from debug print statement. The variable is unitialized and can change values every time executable is run.
- bugfix: unallocated variables
-
Nudging
- Update nudging_io to NF90 API
- Since NetCDF subroutines have been converted to nf90 we can remove the 'include netcdf.inc' statements
-
Config
- config debug: fixes in read_crocus_namelist subroutine
TESTS CONDUCTED:
- Currently testing hydro's WRF testcase on new CMake and old Makefile build systems. PR will be draft until fully tested.
RELEASE NOTE: Hydro has added CMake build option. General bug fixes and code cleanup. MPI routine made more efficient. All hydro file suffixes are now .F90.
LIST OF MODIFIED FILES:
M CMakeLists.txt
A hydro/CMakeLists.txt
M hydro/CPL/WRF_cpl/Makefile
R094 hydro/CPL/WRF_cpl/module_wrf_HYDRO.F hydro/CPL/WRF_cpl/module_wrf_HYDRO.F90
R096 hydro/CPL/WRF_cpl/module_wrf_HYDRO_downscale.F hydro/CPL/WRF_cpl/module_wrf_HYDRO_downscale.F90
R093 hydro/CPL/WRF_cpl/wrf_drv_HYDRO.F hydro/CPL/WRF_cpl/wrf_drv_HYDRO.F90
A hydro/Data_Rec/CMakeLists.txt
M hydro/Data_Rec/Makefile
D hydro/Data_Rec/module_RT_data.F
R069 hydro/Data_Rec/module_gw_gw2d_data.F hydro/Data_Rec/module_RT_data.F90
A hydro/Data_Rec/module_gw_gw2d_data.F90
R099 hydro/Data_Rec/module_namelist.F hydro/Data_Rec/module_namelist.F90
A hydro/Data_Rec/module_namelist_inc.F90
A hydro/Data_Rec/module_rt_inc.F90
A hydro/Debug_Utilities/CMakeLists.txt
M hydro/Debug_Utilities/Makefile
R100 hydro/Debug_Utilities/debug_dump_variable.F hydro/Debug_Utilities/debug_dump_variable.F90
A hydro/HYDRO_drv/CMakeLists.txt
M hydro/HYDRO_drv/Makefile
R099 hydro/HYDRO_drv/module_HYDRO_drv.F hydro/HYDRO_drv/module_HYDRO_drv.F90
A hydro/IO/CMakeLists.txt
M hydro/IO/Makefile
R100 hydro/IO/netcdf_layer.f90 hydro/IO/netcdf_layer.F90
A hydro/MPP/CMakeLists.txt
R094 hydro/MPP/CPL_WRF.F hydro/MPP/CPL_WRF.F90
M hydro/MPP/Makefile
R099 hydro/MPP/hashtable.F hydro/MPP/hashtable.F90
R100 hydro/MPP/module_mpp_GWBUCKET.F hydro/MPP/module_mpp_GWBUCKET.F90
R100 hydro/MPP/module_mpp_ReachLS.F hydro/MPP/module_mpp_ReachLS.F90
D hydro/MPP/mpp_land.F
A hydro/MPP/mpp_land.F90
A hydro/OrchestratorLayer/CMakeLists.txt
M hydro/OrchestratorLayer/Makefile
R099 hydro/OrchestratorLayer/config.f90 hydro/OrchestratorLayer/config.F90
R098 hydro/OrchestratorLayer/io_manager.f90 hydro/OrchestratorLayer/io_manager.F90
R094 hydro/OrchestratorLayer/orchestrator.f90 hydro/OrchestratorLayer/orchestrator.F90
A hydro/Routing/CMakeLists.txt
M hydro/Routing/Makefile
R099 hydro/Routing/Noah_distr_routing.F hydro/Routing/Noah_distr_routing.F90
R100 hydro/Routing/Noah_distr_routing_overland.F hydro/Routing/Noah_distr_routing_overland.F90
R100 hydro/Routing/Noah_distr_routing_subsurface.F hydro/Routing/Noah_distr_routing_subsurface.F90
A hydro/Routing/Overland/CMakeLists.txt
M hydro/Routing/Overland/Makefile
R099 hydro/Routing/Noah_distr_routing.F hydro/Routing/Noah_distr_routing.F90
R100 hydro/Routing/Noah_distr_routing_overland.F hydro/Routing/Noah_distr_routing_overland.F90
R100 hydro/Routing/Noah_distr_routing_subsurface.F hydro/Routing/Noah_distr_routing_subsurface.F90
A hydro/Routing/Overland/CMakeLists.txt
M hydro/Routing/Overland/Makefile
R100 hydro/Routing/Overland/module_overland.F hydro/Routing/Overland/module_overland.F90
R100 hydro/Routing/Overland/module_overland_control.F hydro/Routing/Overland/module_overland_control.F90
R098 hydro/Routing/Overland/module_overland_mass_balance.F hydro/Routing/Overland/module_overland_mass_balance.F90
R100 hydro/Routing/Overland/module_overland_routing_properties.F hydro/Routing/Overland/module_overland_routing_properties.F90
R100 hydro/Routing/Overland/module_overland_streams_and_lakes.F hydro/Routing/Overland/module_overland_streams_and_lakes.F90
R100 hydro/Routing/Overland/overland_tests.F hydro/Routing/Overland/overland_tests.F90
A hydro/Routing/Reservoirs/CMakeLists.txt
A hydro/Routing/Reservoirs/Level_Pool/CMakeLists.txt
M hydro/Routing/Reservoirs/Level_Pool/Makefile
R100 hydro/Routing/Reservoirs/Level_Pool/module_levelpool.F hydro/Routing/Reservoirs/Level_Pool/module_levelpool.F90
R100 hydro/Routing/Reservoirs/Level_Pool/module_levelpool_properties.F hydro/Routing/Reservoirs/Level_Pool/module_levelpool_properties.F90
R100 hydro/Routing/Reservoirs/Level_Pool/module_levelpool_state.F hydro/Routing/Reservoirs/Level_Pool/module_levelpool_state.F90
R100 hydro/Routing/Reservoirs/Level_Pool/module_levelpool_tests.F hydro/Routing/Reservoirs/Level_Pool/module_levelpool_tests.F90
M hydro/Routing/Reservoirs/Makefile
A hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/CMakeLists.txt
M hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/Makefile
R100 hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_persistence_levelpool_hybrid.F hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_persisten\
ce_levelpool_hybrid.F90
R100 hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_persistence_levelpool_hybrid_properties.F hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_p\
ersistence_levelpool_hybrid_properties.F90
R100 hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_persistence_levelpool_hybrid_state.F hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_p\
ersistence_levelpool_hybrid_state.F90
R100 hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_persistence_levelpool_hybrid_tests.F hydro/Routing/Reservoirs/Persistence_Level_Pool_Hybrid/module_p\
ersistence_levelpool_hybrid_tests.F90
A hydro/Routing/Reservoirs/RFC_Forecasts/CMakeLists.txt
M hydro/Routing/Reservoirs/RFC_Forecasts/Makefile
R100 hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts.F hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts.F90
R100 hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_properties.F hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_properties.F90
R100 hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_state.F hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_state.F90
R100 hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_tests.F hydro/Routing/Reservoirs/RFC_Forecasts/module_rfc_forecasts_tests.F90
R100 hydro/Routing/Reservoirs/module_reservoir.F hydro/Routing/Reservoirs/module_reservoir.F90
R100 hydro/Routing/Reservoirs/module_reservoir_read_rfc_time_series_data.F hydro/Routing/Reservoirs/module_reservoir_read_rfc_time_series_data.F90
R100 hydro/Routing/Reservoirs/module_reservoir_read_timeslice_data.F hydro/Routing/Reservoirs/module_reservoir_read_timeslice_data.F90
R100 hydro/Routing/Reservoirs/module_reservoir_utilities.F hydro/Routing/Reservoirs/module_reservoir_utilities.F90
R100 hydro/Routing/Reservoirs/reservoir_tests.F hydro/Routing/Reservoirs/reservoir_tests.F90
A hydro/Routing/Subsurface/CMakeLists.txt
M hydro/Routing/Subsurface/Makefile
R100 hydro/Routing/Subsurface/module_subsurface.F hydro/Routing/Subsurface/module_subsurface.F90
R100 hydro/Routing/Subsurface/module_subsurface_grid_transform.F hydro/Routing/Subsurface/module_subsurface_grid_transform.F90
R100 hydro/Routing/Subsurface/module_subsurface_input.F hydro/Routing/Subsurface/module_subsurface_input.F90
R100 hydro/Routing/Subsurface/module_subsurface_output.F hydro/Routing/Subsurface/module_subsurface_output.F90
R100 hydro/Routing/Subsurface/module_subsurface_properties.F hydro/Routing/Subsurface/module_subsurface_properties.F90
R100 hydro/Routing/Subsurface/module_subsurface_state.F hydro/Routing/Subsurface/module_subsurface_state.F90
R098 hydro/Routing/Subsurface/module_subsurface_static_data.F hydro/Routing/Subsurface/module_subsurface_static_data.F90
R100 hydro/Routing/Subsurface/subsurface_tests.F hydro/Routing/Subsurface/subsurface_tests.F90
R097 hydro/Routing/module_GW_baseflow.F hydro/Routing/module_GW_baseflow.F90
R099 hydro/Routing/module_HYDRO_io.F hydro/Routing/module_HYDRO_io.F90
R088 hydro/Routing/module_HYDRO_utils.F hydro/Routing/module_HYDRO_utils.F90
R098 hydro/Routing/module_NWM_io.F hydro/Routing/module_NWM_io.F90
R099 hydro/Routing/module_NWM_io_dict.F hydro/Routing/module_NWM_io_dict.F90
R100 hydro/Routing/module_RT.F hydro/Routing/module_RT.F90
R094 hydro/Routing/module_UDMAP.F hydro/Routing/module_UDMAP.F90
R099 hydro/Routing/module_channel_routing.F hydro/Routing/module_channel_routing.F90
R100 hydro/Routing/module_date_utilities_rt.F hydro/Routing/module_date_utilities_rt.F90
R097 hydro/Routing/module_gw_gw2d.F hydro/Routing/module_gw_gw2d.F90
R099 hydro/Routing/module_lsm_forcing.F hydro/Routing/module_lsm_forcing.F90
R093 hydro/Routing/module_noah_chan_param_init_rt.F hydro/Routing/module_noah_chan_param_init_rt.F90
R099 hydro/Routing/module_reservoir_routing.F hydro/Routing/module_reservoir_routing.F90
M hydro/arc/Makefile.NoahMP
A hydro/nudging/CMakeLists.txt
M hydro/nudging/Makefile
R100 hydro/nudging/module_date_utils_nudging.F hydro/nudging/module_date_utils_nudging.F90
R090 hydro/nudging/module_nudging_io.F hydro/nudging/module_nudging_io.F90
R085 hydro/nudging/module_nudging_utils.F hydro/nudging/module_nudging_utils.F90
R100 hydro/nudging/module_stream_nudging.F hydro/nudging/module_stream_nudging.F90
A hydro/utils/CMakeLists.txt
M hydro/utils/Makefile
R100 hydro/utils/module_hydro_stop.F hydro/utils/module_hydro_stop.F90
R100 hydro/utils/module_version.F hydro/utils/module_version.F90
The results of regression test from the last commit:
Test Type | Expected | Received | Failed
= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Number of Tests : 23 24
Number of Builds : 60 57
Number of Simulations : 158 150 0
Number of Comparisons : 95 86 0
Failed Simulations are:
None
Which comparisons are not bit-for-bit:
None
PR is ready for review. Tested Intel and GNU CMake builds. Needed to make updates to get both working but now both build and run. Repeat runs of the Intel compiler produced identical results for the testcase, same was true for GNU. The testcase was the CO Front Range, the model time was 12 hours, and was run with four MPI ranks.
@scrasmussen @rcabell The regression test report didn't come back after the last commit. Can you make some trivial change to trigger another test?
@scrasmussen Again, I have not received any regression test report from the last couple of commits. Note each commit to your branch should trigger a regression test.
@scrasmussen Again, I have not received any regression test report from the last couple of commits. Note each commit to your branch should trigger a regression test.
@weiwangncar thanks for pointing this out again. I've been making commits so I wonder why it isn't triggering? I try clicking on the Details link but it times out. Maybe somewhere I don't have the permissions to kick off the regression tests properly?
@scrasmussen I don't think you can trigger a test by clicking on 'Details' link. The only way to do so is to make a trivial change in a .F file, a Makefile, etc.. I'm not sure a change in a .txt will do it, since we've not had .txt files up to the availability of Cmake.
@scrasmussen Thanks for making a trivial change. Here is the latest regression test results:
Test Type | Expected | Received | Failed
= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Number of Tests : 23 24
Number of Builds : 60 57
Number of Simulations : 158 150 0
Number of Comparisons : 95 86 0
Failed Simulations are:
None
Which comparisons are not bit-for-bit:
None
Converted to draft because I need to bring over a few CMake build changes to the nudging configuration and double check everything works correctly. Will make those changes tomorrow.
@scrasmussen Thanks for making a trivial change. Here is the latest regression test results:
@weiwangncar thanks for your help in figuring out how to trigger the regressions tests!
If I'm not mistaken, this is the last PR that we're waiting on before creating the release-v4.6.0 branch. Do we have a sense for how soon we'll be able to get this PR finished off?
@mgduda @islas I think so. Is there still revision we expect from this PR? Remember we can still tweak and fix the code after the cut-off.
@weiwangncar I have one revision request left which should fix MPI linking issues
@scrasmussen @rcabell Do you have an estimate for when this PR will be out of draft mode and ready for final review? Is it just a resolution to the MPI linking issue pointed out by @islas that needs to be resolved?
@mgduda So sorry, I had missed these comments! I believe I made a change that will fix the MPI linking issue that @islas found? I wasn't able to replicate that behavior but on Derecho the change built and gave identical results to the previous version.
I am happy to move this from draft to ready for review but just want to give you a heads up. A user of WRF-Hydro has been having an issue, see #741, on Derecho where long runs with a large domain crash from running out of memory. Possibly have a fix but still testing and those changes aren't in this hydro version. Not sure how you want to handle this? Hoping to have a fix by start of next week.
@scrasmussen I tested out the latest commit updates (d8bb163) but still got the same error:
[ 98%] Linking Fortran static library libWRF_Core.a
[ 98%] Built target WRF_Core
Scanning dependencies of target hydro_wrf_cpl
[ 98%] Building Fortran object hydro/CPL/WRF_cpl/CMakeFiles/hydro_wrf_cpl.dir/module_wrf_HYDRO.F90.o
/home/aislas/wrf-model/wrf_alt/hydro/CPL/WRF_cpl/module_wrf_HYDRO.F90:25:9:
25 | use mpi
| 1
Fatal Error: Cannot open module file ‘mpi.mod’ for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [hydro/CPL/WRF_cpl/CMakeFiles/hydro_wrf_cpl.dir/build.make:88: hydro/CPL/WRF_cpl/CMakeFiles/hydro_wrf_cpl.dir/module_wrf_HYDRO.F90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2114: hydro/CPL/WRF_cpl/CMakeFiles/hydro_wrf_cpl.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
The target hydro_wrf_cpl currently doesn't include the MPI includes and add_dependencies() unfortunately doesn't carry any target transitive properties like target_link_libraries()
@islas thanks for testing it and your detailed message! I added the Fortran MPI include directories to the hydro_wrf_cpl target. I did a verbose build to double check it was including the MPI module path, so I hope it is fixed now.
@scrasmussen Do you mind if we pull this out of draft mode to merge in?
@mgduda @weiwangncar Do you two have any thoughts on pulling this in given the NCAR/wrf_hydro_nwm_public#741 issue?
My understanding is the issue is part of WRF Hydro itself, but as this PR adds the capability to build it analogous to the standalone repo using CMake this may have the same issue the standalone repo is seeing.
Since the makefile method still exists and the issue is probably more tied to the underlying code than build method I don't think it should be a roadblock.
@scrasmussen or @rcabell Please feel free to correct any of that if my understanding was mistaken.
Agreed - the underlying issue in #741 is almost certainly unrelated to the build system. This can proceed independently.
On Mar 15, 2024, at 12:09 PM, Anthony Islas @.***> wrote:
My understanding is the issue is part of WRF Hydro itself, but as this PR adds the capability to build it analogous to the standalone repo using CMake this may have the same issue the standalone repo is seeing.
Since the makefile method still exists and the issue is probably more tied to the underlying code than build method I don't think it should be a roadblock.
@scrasmussen https://github.com/scrasmussen or @rcabell https://github.com/rcabell Please feel free to correct any of that if my understanding was mistaken.
— Reply to this email directly, view it on GitHub https://github.com/wrf-model/WRF/pull/2009#issuecomment-2000187718, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHEYLT2WZS2RDN6BTITZ7TTYYM2NRAVCNFSM6AAAAABDRSJIMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBQGE4DONZRHA. You are receiving this because you were mentioned.
