betterscientificsoftware
Possible Blog Article: No code offered to users should NOT be regularly tested
In a short conversation with @bartlettroscoe, we talked about having code that is used as examples/tutorials and to train others in how to use the software or training others in how to develop the software.
It is hard to write code aimed at testing all the nooks and cranies of a software package that is also a useful from an end-user training perspective.
Nonetheless, @bartlettroscoe observed that the worst possible code to NOT regular test is that which you give to users as examples/tutorials.
We should find resources and/or write content about this topic.
Just a comment that the title of this issue is too convoluted to use as the title of the article. Maybe something more like "Always test your examples and tutorials".
@markcmiller86, note that we have gravitated to two types of executable "tests" in Trilinos. The first are hard-core tests that are meant to test and provide feature, line, and branch coverage. The second are "examples" that are designed to demonstrate how the software works but also have some pass/fail criteria to ensure that the code is working correctly. It is part of this "example" code that is pulled into documentation. I have done this with both C++ and CMake code. (It is easier to extract code fragments from C++ to put in documentation using a tool like Doxygen. There is no such tool for CMake.)
It is part of this "example" code that is pulled into documentation.
If by pulled into, you mean that what is actually rendered in the hosted documentation is the actual code that is also being tested (instead of a cut-n-pasted copy of that code), YES, THAT IS A TERRIFIC WAY TO DO BUSINESS. We are doing more and more of this with VisIt.
It is part of this "example" code that is pulled into documentation.
If by pulled into, you mean that what is actually rendered in the hosted documentation is the actual code that is also being tested (instead of a cut-n-pasted copy of that code), YES, THAT IS A TERRIFIC WAY TO DO BUSINESS.
Yes, there is no manual copy and paste. All extraction of example code is pulled from CI tested code.
We are doing more and more of this with VisIt.
👍