Support for Incremental tests by selectively disabling shuffling

[Querela]

Is it possible to use pytest-randomly but selectively disable shuffling of functions in classes?

Similar to https://github.com/jbasko/pytest-random-order#disable-shuffling-in-module-or-class This could allow incremental tests, as described in https://docs.pytest.org/en/stable/example/simple.html#incremental-testing-test-steps but still allow some randomness. In incremental tests where previous functions may depend on prior steps (organized in functions in a class) random order would expectedly fail.

[adamchainz]

I'd be open to adding markers like pytest-random-order. It would also help with rollout of pytest-randomly on a large codebase, one can mark off the "unsafe to shuffle" modules/classes without needing to fix all their interdependencies immediately.

[Querela]

I saw that there is a CLI option to disable shuffling on a session scope when I looked at the code. For module and class scope it might be possible to just check for a custom mark, e.g. 'no-shuffle-mark' in item.keywords, and disable the shuffling of test items before adding them to the collection. That should be 2-4 locations in the code (just before the random.shuffle call). When annotating a module (pytestmark), the marker should be recorded in a class and function items if I remember correctly, so the checks can be the same. The only issue might be marked functions (which makes no sense but could disable shuffling unintentionally for a larger scope depending on how checks are implemented).

Something along:

random.shuffle(current_items)
# -->
do_shuffle = not all(("no-shuffle-mark" in item.keywords) for item in current_items)
if do_shuffle:
    random.shuffle(current_items)

Maybe item.iter_markers(name="...") if "..." in item.keywords does not work?