Pytest plugin for automating test parallelism checks

[tonybaloney]

Hey,

We just published a pytest plugin to automate running a test suite in parallel since this isn't possible with pytest-xdist.

https://pypi.org/project/pytest-freethreaded/

The pattern is loosely based on guidance in this repo.

We also discovered when running a couple of test suites some pretty nasty bugs, including a segfault in Pillow https://github.com/tonybaloney/pytest-freethreaded/issues/11

Think it could be worth mentioning the plugin on the site (happy to submit a PR).

It's not designed to be run in CI since

1. It would be really slow
2. Concurrency bugs are not necessarily the fault of no-gil

Instead, I expect maintainers to run this a few times to help find any really nasty bugs.

[ngoldbaum]

@andfoy has been working on his own pytest plugin: https://github.com/Quansight-Labs/pytest-run-parallel

Maybe there's some overlap?

[rgommers]

It looks like it's time to document both pytest-freethreaded and pytest-run-parallel in https://py-free-threading.github.io/debugging/, since both plugins are uncovering new issues. The existing content in that section of the docs essentially describes doing manually what these pytest plugins now do automatically.

@andfoy would you be able to update the docs for this?

[rgommers]

This was indeed added to the docs a few weeks ago: https://py-free-threading.github.io/porting/#validating-thread-safety-with-testing.

We're now up to 3 such testing tools, unittest-ft was added in a follow-up. It's proving to be a very useful technique I think.

Thanks for the issue and the plugin @tonybaloney!