Checkin Test Script Feature Request: Ignore tests that were failing before our changes

[csiefer2]

As discussed at TUG16, a useful TriBITS feature would be a way to tell the checkin test script to ignore tests that were failing before the changes represented by my commits.

[bartlettroscoe]

Yup, that has been in the backlog for a while now. See:

• https://docs.google.com/document/d/1WKs8rJhI3037yKGnEVMhIx9dPN7a7uFRM5isdNAhAXI/edit#bookmark=id.e6bef1l7mks
• https://docs.google.com/document/d/1WKs8rJhI3037yKGnEVMhIx9dPN7a7uFRM5isdNAhAXI/edit#bookmark=id.x0u2uu57d9w1

which states:

• Add option --compare-tests-to=<base-testing-dir> to allow comparing test results to another testing directory and set of test results. If no new tests are failing, then success is recorded which would allow the push.
• Add option --compare-to-control-build which would use another set of git repos and build dirs to build and test the same version of the code, minus local changes (use git merge-base <tracking-branch> <current-branch> for the reference version) and then use the --compare-tests-to=<base-testing-dir> option to determine pass/fail. You would only have to do build cases with one or more failing tests and you would only have to enable packages with failing tests. Therefore, this “control build” could be much cheaper than the main build.

[csiefer2]

Something like that would do exactly what I've suggested.

[bartlettroscoe]

To be safe, we would also need another option like --allowed-failed-control-tests=<M> that would restrict the number of existing failing tests that would be allowed in order to give the okay to push. For example, if just 5 or 10 tests are already failing, then that might be okay but if 1000 are already failing, then it is likely not safe to push because you may be breaking those tests in a new way and never know it. I would say for Trilinos the allowed number should be no higher than 5 or 10.

[csiefer2]

That sounds very reasonable.