pytest-test-groups icon indicating copy to clipboard operation
pytest-test-groups copied to clipboard

Grouping tests by files instead of items

Open ishefi opened this issue 4 years ago • 0 comments

Thanks for this plugin! We are using it and it helped lowering our pipeline running times from ~60m to ~7m.

However, I found out that failures of unittets tend to be across a small number of files. Since unitests of the same file are devided between multiple (usually all) groups, multiple groups are failing due to a bug that stems in the same area.

This PR adds an option to split by files rather than by items, making the investigation of failures much easier.

Couple of notes, as I'm a bit afraid this PR might not be in the spirit of this project:

  1. If the number of files is smaller than total groups some groups will run no tests.
  2. Some jobs might take longer than others, as there is no way to guarantee tests will be equally divided between jobs. The division is done using a greedy algorithm which tries to split the tests so each job runs about the same number of tests, but in extreme cases some jobs will work much harder than others (e.g.: only two test files need to run: file1 with 1000 tests, file2 with 2 tests).

Thank you again!

ishefi avatar May 20 '20 18:05 ishefi