fortran-lang
Skip slow tests
Description
pytest has decorators which allow for a variety of conditional executions. It would be useful to have a way of marking slow tests, or other kinds of conditional test runs.
Possible Solution
Technically each test can check environment variables (as noted by @certik), but it would be nicer to have this at the test runner level.
Additional Information
N/A.
The use case we have is generating data for plots into an article. Some of the convergence studies might take a longer time (say several minutes) and we would like the default "fpm test" to finish in a few seconds. One approach is to have "slow tests" that are not executed by default. We can then internally run the full convergence study in the slow test, and a faster version of the same study in the fast test (to ensure that the study works).
I tend to break my tests into a confidence test that is very quick but exercises a few key features, a collection of short tests that do coverage, and what I usually call benchmark tests that are long-running. When I have moved those to fpm I have been breaking them up by name with the globbing feature like "run test 'bench_*', run test 'general_*' and run test confidence; and in one case I made the test program a program that takes parameters that select which directory to run and put the other test programs into directories and did not find any of those quite hit the mark, so I think I will take a look at how pytest does it. Any other examples of models to try? I was thinking about lists of tests or directories that you could give a name to in the fpm.toml file, and then do something like "fpm run test -type plot" or "fpm run test -type benchmark", and earlier I had a PR that allowed for a "y/n" reply to a prompt that no one else seemed to like so I didn't pursue it.
