pytest-split
pytest-split copied to clipboard
fix least duration algo with random ordering
Description
Hi!
I found a rather critical bug in the least duration algorithm when running together with random ordering and when durations do not exist. The issue arises from the way pytest represents a test case item. It uses only the test case name. In the least duration algorithm you use the string representation of the test item to sort the test cases to ensure deterministic groups even when random order is applied. Due to the fact that only the name is used for ordering, the split groups become undeterministic (depending on the initial order) in those cases when there are multiple tests with same name in different modules and durations does not exist. This leads to repeated and skipped tests when used in CI for example.
A very simple solution would be to use the nodeid to sort the tests which I tested and implemented in this PR. I hope you find it OK but if modifications needed I am happy to do that.
Checklist
- [✅ ] Tests covering the new functionality have been added
- [❌ ] Documentation has been updated OR the changes are too minor to be documented
- [✅ ] The Changes are listed in the
CHANGELOG.mdOR the changes are insignificant