Inconsistent Output Baseline Comparisons
Description
After fixing a number of failing tests in the recent weeks, I came across some inconsistencies in the baseline test cases. Essentially, all of the baseline comparisons that compare an image without the name OutputBaseline, are incorrectly comparing results. An example can be found in the AddNoiseToBinaryImageTestBaselineComparison, which produces:
/Users/mseng_l/ITKExamples-build/bin/ImageCompareCommand "--test-image"
"/Users/mseng_l/ITKExamples-build/src/Core/Common/AddNoiseToBinaryImage/AddNoiseToBinaryImage.png" "--baseline-image"
"/Users/mseng_l/ITKExamples-build/src/Core/Common/AddNoiseToBinaryImage/AddNoiseToBinaryImage.png"
As shown above, the baseline comparison is not comparing the example's output to the baseline image, but rather the baseline image to itself, which will always yield a passing result.
This and many others will likely require one of the following changes:
- [ ] Remove baseline comparisons for examples that do not produce a consistent result (i.e.
AddNoiseToBinaryImagedue toMersenneTwisterRandomVariateGenerator) - [ ] Rename baseline images to
OutputBaselinefor examples that consistently produce and save their output - [ ] Implement
ImageFileWriterfor examples that do not save their output. Subsequently update the baseline comparison for that example, upload the image toGirderand finally update theDocumentation.rst
We can set a deterministic seed in the examples which have some pseudo-randomization. Together with setting the number of work units to 1 it should produce a deterministic output.