single executable multiple data test or regression

[kurtsansom]

context: Converting a regression test to use pfunit from fruit, and the original test is not really unit sized compares a stored "analytical" solution to the output of the program.

I want to generate one executable and then manually add a ctest for each data set. Is there a simple way to do this?

I initially went down the path of using add_pfunit_sources.cmake but then there appears to be a lot of cmake configuration stuff that has to be added to get it to work. There appears to be one reference in the pFUnit tests to add_pfunit_sources.cmake.

Are there better options out there for regression testing? or a suggested way forward to do regression testing with pfunit?

Most things I have seen end up being home grown, much like the code I am refactoring. I have used ats in the past which is a higher level framework that was designed for running large test suites on HPC systems, but is more simple because it relies on an executable returning 0 or not 0 to determine pass fail criteria.

[tclune]

Hmm. Yes, one of the hard aspects of a clean testing framework is that it can be difficult to "inject" information from outside.

One hackish thing that you could do is to generate a small Fortran file for each such case. It would contain a simple procedure that returns the (hardwired) name of the data file containing the "analytical" solution. The test itself would call this procedure to find the data. I'll leave the cmake details to you, but they should be reasonably straightforward.

Another option that might work would be to tie the name of the ctest executable to the filename with the data. The "extra" initialize could then query its own name (Fortran command line arguments) to derive the name of the input file. This could be then stored in a global variable that is then accessed by the test itself. Still a bit kludgy, but at least there are not extra files being generated.

Another variant would be to just encode an integer rather than a string with the previous mechanism. You could then have a file containing a list of data file names and the test would use the integer to determine which file to open.

And for the other readers ... a pFUnit parameterized test can handle this cleanly. But it would bundle all of the tests under a single ctest, which presumably has downsides that have not been articulated in this thread. (Long running tests, unstable cases ...)