Feature enhancement to allow turning plugin on and off

[clintonsteiner]

Description

Hello,

Using this plugin occasionally while writing new tests to prevent flakiness, but old tests still need to run without the random order. Currently accomplishing this via pytest.ini which includes -p no:randomly and a separate one you use via -c pytest-randomly.ini

This feels rather clunky, I'd like to propose creating a pytest command line argument to turn the plugin off as needed The other option is to create a bash variable PYTEST_USE_RANDOMLY which defaults to true, but can be flipped on and off as you go

I'm more than happy to make this change, but wanted to run it by the community before opening the PR.

[adamchainz]

to turn the plugin off as needed

This is hard to do, the plugin acts at different points in pytest's test lifecycle: collecting tests, then running modules, classes, and functions.

To force order for specific tests, you can use pytest-order, which even documents usage with pytest-randomly. Try that for your legacy tests?

[clintonsteiner]

Hi @adamchainz thank you for maintaining this plugin and taking the time to respond to my issue.

The project I'm using this for is massive, and adding markers to each test is not feasible. Outside of being difficult to do, do you have any objection to the ability to turn this plugin off but leave it installed via a command line argument? If would be open to the idea I will fork the repo and begin work, but didn't want to work on something that was fundamentally against the project nature.

From what I can tell, I think the change would need to occur inside https://github.com/pytest-dev/pytest-randomly/blob/main/src/pytest_randomly/init.py More specifically within the hook function pytest_collection_modifyitems

Let me know what your thoughts are and thanks once again.

[adamchainz]

I think your proposal is too niche to include in pytest-randomly right now. If others +1 this issue, maybe.

Also, you may be able to achieve what you want with a custom plugin in your project. pytest plugins are very flexible and it should be possible to make one that disables pytest-randomly by default - perhaps by telling pytest to disable the plugin, or perhaps by activating the --randomly-dont-reset-seed and --randomly-dont-reorganize options. Give that a try?

[MetRonnie]

First time using this plugin and my initial thoughts are: I want this to be opt-in rather than opt-out

[clintonsteiner]

Hey @adamchainz any chance you'd reconsider this? I'd be happy to do the pr if you're open to this idea. Would also be fine putting it in an optin config file somewhere if you'd prefer that

[adamchainz]

I am not really convinced that we need any new mechanism here for opting in. We already have these:

1. Disable the plugin for some test runs with -p no:randomly.
2. Enable the plugin only for some test runs by adding -p no:randomly in your pytest settings then adding -p randomly on the opt-in runs. (Or just install pytest-randomly for those runs.)
3. Disable features with --randomly-dont-reset-seed or --randomly-dont-reorganize.

It doesn't seem necessary to me to introduce another mechanism and to maintain that for a long time. The plugin is only a few hundred lines long with rare changes. If you need extreme customization, you can also copy the relevant parts into your project's conftest.py.

[alexandrul]

-p no:randomly is not working in some corner cases: https://github.com/pytest-dev/pytest-randomly/issues/647