CBICA
Regression Testing
Is your feature request related to a problem? Please describe. Unit Tests should also compare the changes in results (if any) across various compilers and when 3rd party library versions change.
Describe the solution you'd like One example would be here: https://github.com/SCOREC/core/wiki/Regression-Testing
Describe alternatives you've considered Another easy alternative is to generate and store the results generated using a single run and use that as the benchmark (once the results are validated by the appropriate researcher).
Additional context Other reading:
- https://lhncbc.nlm.nih.gov/publication/lhncbc-2009-096
- https://github.com/SRombauts/cmake-basics (the CTest here as an example of using "Regular Expression" for testing)
I've been taking a look at this for the past couple days and have been considering some possible options, it looks like regression tests would need to have a manual element to them since I don't think there is a way to get data off of Travis (unless we deploy, but that's intended for actual versions of CaPTk). CDash also is a program that would need to be ran/hosted somewhere, so having a dashboard would be nice but it would involve setting up.
Right now my thoughts are to make use of make test as a basis for regressions. Before any releases or patches, developers can use the output of make test as a baseline to compare against older regressions. The output includes test names, time it took to complete, and what types of failures are encountered if any. If we keep it local, we can use large images from the testing data to help researchers compare differences in images and identify regression issues. In addition, since make test can be ran on any commit of CaPTk, regressions against specific commits can be made. I think a set of tools could be made to make these processes easier, and it probably wouldn't be too hard to get something working on linux, but we'd ideally want a cross-platform solution. This is very manual, but some benefits of this is that it should encourage application developers to add their own tests and makes doing regressions an active thing developers should care about.
My fears are if we move this all to the CI we either 1) won't be able to save regressions 2) using meaningful data will take too long to test (already pushing it on windows and osx) or 3) the use of meaningful data will add a non-trivial amount of time to travis to download it which may put us over the space limitations.
(Accidentally closed the issue, it's right next to the comment button...)