WRF icon indicating copy to clipboard operation
WRF copied to clipboard

Initial testing framework

Open islas opened this issue 1 year ago • 2 comments

TYPE: enhancement

KEYWORDS: testing, regression, test framework

SOURCE: internal

DESCRIPTION OF CHANGES: Problem: The current regression suite code is complex, requires maintenance of multiple alternate repositories, and takes involved effort to add a new test making community contribution limited at best. Likewise, the complexity of the system reduces the likelihood of independent local testing of changes, leading to a development cycle of one-off commits done to reinvoke testing to see if meaningful commits fix the issues.

Solution: This new proposed regression suite addresses these shortcomings in a number of discrete ways:

  1. Modularize the testing framework to an generalized independent repo usable by any repo seeking to set up tests that can run locally, on HPC systems, and within any CI/CD framework
  2. Write WRF-specific test scripts inside the WRF repo and in a manner that does not rely on specific layouts/hardware/etc. so long as WRF can compile and run on intended system (i.e. able to be run locally)
  3. Write CI/CD tests in a simple and generally CI/CD framework-agnostic method where definitions of these also reside within the WRF repo
  4. Utilize HPC resources in a safe manner to increase breadth of testing to allow testing of many more compilers and on similar hardware to the general use case of WRF

As a first pass at demonstrating this solution, this PR implements a simple set of compilation tests using GNU x86 configurations testing serial, sm, dm, and sm+dm selections.

TESTS CONDUCTED:

  1. Testing of this github workflow was done in a separate fork also testing on Derecho. Both positive and negative tests were used to demonstrate respective output usefulness.

RELEASE NOTE: Introduce a modularized testing framework that allows testing locally and natively on HPC systems that lives within the WRF repository

islas avatar Aug 09 '24 21:08 islas

I'm using the approach we're using in MPAS to setup testing with a very limited minimal setup (simple compilation tests) at first to get something started.

The idea would be to then gradually translate the current tests to a usable format by this framework.

islas avatar Aug 09 '24 21:08 islas

The regression test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

weiwangncar avatar Aug 10 '24 02:08 weiwangncar