yellowbrick
yellowbrick copied to clipboard
Update tests to use windows_tol instead of conditional tolerances
Now that #864 has been merged in, we have an argument, windows_tol
, for assert_images_similar
, which will allow us to keep our tests robust for our most common operating systems and to continue to provide CI coverage for other operating systems. Currently, many of our image comparison tests use conditional tolerances to account for slight variations in fonts and colors between Mac and Windows operating systems. This issue requests that we update our visual tests to ensure we are using the new windows_tol
instead of conditional tolerances like xfails
.
I analyzed the xfails and non-xfail, conditional tols based on win32
in https://github.com/DistrictDataLabs/yellowbrick/issues/393#issuecomment-500184165
The upshot is that based on the current state of the develop branch, 2/3 of the xfails or conditional tols can be removed completely, and the remaining 1/3 are of the type IS_WINDOWS_OR_CONDA. These can be resolved by
- Making the conda xfails pass instead by simply substituting the testing version of matplotlib in the Windows conda and Linux conda builds, through the dependencies. This will take care of Freetype differences, as discussed in #892.
- Updating the IS_WINDOWS_OR_CONDA xfails in #892 to instead use windows_tol.
windows_tol
would then apply only to Windows PyPI (just my term for non-conda Windows, there may be a better one).
In the long-term, all Freetype differences could potentially be resolved for not only the existing visualizers but all future visualizers across all platforms and distributions if the matplotlib team were open to creating testing-focused pip package(s) similar to the conda package.
This would significantly decrease the need to spend time on tols.
If the milestone PR's reach a point that is convenient, please let me know if you would like me to coordinate with @ndanielsen on a PR or PRs to eliminate the conditional xfails and shift to windows_tols.
FYI, the plan is to search through for sys.platform
in the tests directory and change things like this:
tol = (
1.5 if sys.platform == "win32" else 1.0
) # fails with RMSE 1.409 on AppVeyor
self.assert_images_similar(oz, tol=tol)
to
# fails with RMSE 1.409 on AppVeyor
self.assert_images_similar(oz, tol=1.0, windows_tol=1.5)