adobe-type-tools
pytest-randomly causes occasional CI failures
While attempting to sync dependencies between dev-requirements.txt and setup.py, I re-introduced pytest-randomly which was listed in setup.py but not in dev-requirements.txt. When running the test suite in CI, some test runs failed on the test_hashmap_old_version and test_hashmap_no_version tests (but were occasionally successful upon running again). I concluded that the random test execution order introduced by pytest-randomly was the cause, and removed it. This should be investigated as it suggests that there is some precondition that is getting set when the tests are run in the defined order but which is not always met when run randomly.
The test that failed (test_hashmap_no_version) checks for the presence of a specific logging message.
By default, psautohint filters out duplicate logging messages.
To get rid of the gremlins, you may have to implement something similar to this change throughout the test suite.
A simple way to test this hypothesis would be to disable the dedup filter, and then enable pytest-randomly.
Bottom line is: deduping logging messages is desired when running the tools, but causes problems when running the tests.
I removed some of the duplicate message removal code in otfautohint, so I don't know if this still applies. I'll leave it open.
