pytest-randomly
pytest-randomly copied to clipboard
pytest-dev
Shuffle inside modules only mode
Description
We're trying to transition to randomised tests, and this is a feature that I feel strongly about as a helpful one in the period of transition when you expect lots of failures. This is our project dirs:
/root
- tests/
- - new_app_tests/
- - old_app_tests/
- - unit_tests/
When I run the tests, they're shuffled within the tests/ folder:
tests/new_app_tests/some_tests3.py . . . . . . . . .
tests/old_app_tests/ some_tests4.py. . . . . . .
tests/unit_tests/some_tests22.py . . . . . . . . .
tests/new_app_tests/some_tests10.py . . . . . . . . .
While it would be more convenient for the transition period to have tests shuffled within subdirectories of tests/, like this:
tests/new_app_tests/some_tests3.py . . . . . . . . .
tests/new_app_tests/some_tests2.py . . . . . . . . .
tests/new_app_tests/some_tests1.py . . . .
tests/new_app_tests/some_tests0.py . . . . . . . . .
tests/old_app_tests/ some_tests4.py. . . . . . .
tests/old_app_tests/ some_tests3.py. . . . . . .
tests/old_app_tests/ some_tests1.py. . . . . . .
tests/old_app_tests/ some_tests2.py. . . . . . .
tests/unit_tests/some_tests2.py . . . . . . . . . . . . . . . .
Does this make sense?
What motivates this? Are you seeing a lot of test failures due to shuffling? Did you try the --randomly-dont-reorganize flag, which disables all re-ordering? Perhaps you could install with that flag, then remove it in a branch and rerun tests many times to find and fix your ordering-related problems.
