New Testing Framework

[islas]

TYPE: enhancement

KEYWORDS: testing, regression, test framework

SOURCE: internal

DESCRIPTION OF CHANGES: Problem: The current regression suite code is complex, requires maintenance of multiple alternate repositories, and takes involved effort to add a new test making community contribution limited at best. Likewise, the complexity of the system reduces the likelihood of independent local testing of changes, leading to a development cycle of one-off commits done to reinvoke testing to see if meaningful commits fix the issues.

Solution: This new proposed regression suite addresses these shortcomings in a number of discrete ways:

1. Modularize the testing framework to an generalized independent repo usable by any repo seeking to set up tests that can run locally, on HPC systems, and within any CI/CD framework
2. Write WRF-specific test scripts inside the WRF repo and in a manner that does not rely on specific layouts/hardware/etc. so long as WRF can compile and run on intended system (i.e. able to be run locally)
3. Write CI/CD tests in a simple and generally CI/CD framework-agnostic method where definitions of these also reside within the WRF repo
4. Utilize HPC resources in a safe manner to increase breadth of testing to allow testing of many more compilers and on similar hardware to the general use case of WRF

LIST OF MODIFIED FILES: A .ci/env/cheyenne.sh A .ci/env/derecho.sh A .ci/env/helpers.sh A .ci/env/hostenv.sh A .ci/env/wsl.sh A .ci/missingTests.md A .ci/tests/SCRIPTS A .ci/tests/build.sh A .ci/tests/runNamelists.sh A .ci/wrf_arw_tests.json A .ci/wrf_chem_tests.json A .ci/wrf_em_ideal_tests.json A .ci/wrf_em_real_tests.json A .github/workflows/ci.yml M .gitignore M .gitmodules M arch/postamble M configure M external/io_netcdf/diffwrf.F90 A hpc-workflows M main/depend.common

TESTS CONDUCTED:

1. All tests were conducted on cheyenne and derecho to replicate as close as possible a one-to-one matching of the current regression suite

RELEASE NOTE: Introduce a modularized testing framework that allows testing locally and natively on HPC systems that lives within the WRF repository

[islas]

Modifications to configure are to be superseded by #1743 Modifications to main/depend.common are to be superseded by #1950

[weiwangncar]

The regression test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None