ITKSphinxExamples icon indicating copy to clipboard operation
ITKSphinxExamples copied to clipboard

Inconsistent Output Baseline Comparisons

Open mseng10 opened this issue 4 years ago • 1 comments

Description

After fixing a number of failing tests in the recent weeks, I came across some inconsistencies in the baseline test cases. Essentially, all of the baseline comparisons that compare an image without the name OutputBaseline, are incorrectly comparing results. An example can be found in the AddNoiseToBinaryImageTestBaselineComparison, which produces:

/Users/mseng_l/ITKExamples-build/bin/ImageCompareCommand "--test-image" 
"/Users/mseng_l/ITKExamples-build/src/Core/Common/AddNoiseToBinaryImage/AddNoiseToBinaryImage.png" "--baseline-image" 
"/Users/mseng_l/ITKExamples-build/src/Core/Common/AddNoiseToBinaryImage/AddNoiseToBinaryImage.png"

As shown above, the baseline comparison is not comparing the example's output to the baseline image, but rather the baseline image to itself, which will always yield a passing result.

This and many others will likely require one of the following changes:

  • [ ] Remove baseline comparisons for examples that do not produce a consistent result (i.e. AddNoiseToBinaryImage due to MersenneTwisterRandomVariateGenerator)
  • [ ] Rename baseline images to OutputBaseline for examples that consistently produce and save their output
  • [ ] Implement ImageFileWriter for examples that do not save their output. Subsequently update the baseline comparison for that example, upload the image to Girder and finally update the Documentation.rst

mseng10 avatar Feb 26 '21 16:02 mseng10

We can set a deterministic seed in the examples which have some pseudo-randomization. Together with setting the number of work units to 1 it should produce a deterministic output.

dzenanz avatar Feb 26 '21 19:02 dzenanz