pytest-xdist causes warnings to be emitted when a unit test uses os.fork()

[zzzeek]

test suite:

import time
import multiprocessing

class TestWhatever:

    def test_thing(self):

        def go():
            time.sleep(2)

        ctx = multiprocessing.get_context("fork")
        proc = ctx.Process(target=go, args=())
        proc.start()

Running as pytest test.py -n1, output:

[classic@framework tmp2]$ pytest test.py -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp2
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
1 worker [1 item]      
.                                                                                                                                                                                                 [100%]
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16078) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 2.19s ======================================================================================

Per the author of this warning, multithreaded code is never safe if it also spawns using fork (see discussion). However I cannot locate any threads running. Here's an extension of the example that lists out threads running, and I can find none that are not the "main" thread:

conftest.py:

# conftest.py
import threading
import os
import logging

logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)


class XDistHooks:
    def pytest_configure_node(self, node):
        for t in threading.enumerate():
            logging.getLogger("main").info(
                f"THREAD FROM MAIN PROCESS {os.getpid()}: {t}\n")


def pytest_configure(config):

    if config.pluginmanager.hasplugin("xdist"):
        config.pluginmanager.register(XDistHooks())

test.py:

# test.py
import time
import os
import multiprocessing
import logging
import threading

logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)

class TestWhatever:

    def test_thing(self):
        for t in threading.enumerate():
            logging.getLogger("main").info(
                f"THREAD FROM CHILD PROCESS {os.getpid()} "
                f"(parent: {os.getppid()}): {t}\n")

        def go():
            time.sleep(10)

        ctx = multiprocessing.get_context("fork")
        proc = ctx.Process(target=go, args=())
        proc.start()

run output:

[classic@framework tmp]$ pytest test.py   -s -p no:logging -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
initialized: 1/1 workerINFO:main:THREAD FROM MAIN PROCESS 16341: <_MainThread(MainThread, started 139752741145472)>

1 worker [1 item]      
INFO:main:THREAD FROM CHILD PROCESS 16342 (parent: 16341): <_MainThread(MainThread, started 139984508341120)>

.
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16342) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 10.17s =====================================================================================

Basically I want to keep using fork() in my test code, since we are running functions inside the tests themselves in processes. Where is pytest-xdist and/or execnet spawning threads exactly (code is pretty opaque) and is this a bug in the python interpreter?

[RonnyPfannschmidt]

Recent xdist+execnet has a workaround to ensure xdist runs on the main thread

[RonnyPfannschmidt]

This is a longstanding issue with execnet

The introduction of execmodels made it possible for pytest to run on non main threads

[zzzeek]

why is the thread not showing in threading.enumerate() ?

[RonnyPfannschmidt]

https://github.com/pytest-dev/execnet/blob/b0b5e4c3391865985f27cbb1e8b20af0f34d9423/src/execnet/gateway_base.py#L155C41-L155C42

seems its using the lowlevel primitive

[zzzeek]

that's what I thought though I didnt know you could do that from pure python. OK! I think I might have even known about this issue at some point so I'm going to note this , thanks

[RonnyPfannschmidt]

i think we should definitively use the high-level ones instead as debugging those is a pain

[RonnyPfannschmidt]

i wonder if there was a specific technical reason for how it ended up with lowlevel primitives for threads

[diranged]

We're using the latest versions of pytest and pytest-xdist but we're still seeing this warning ... and we are occasionally seeing our builds hang. Is this supposed to be resolved?

pytest==8.3.5
pytest-asyncio==0.25.3
pytest-django==4.10.0
pytest-timeout==2.3.1
pytest-xdist==3.6.1

[RonnyPfannschmidt]

With the latest versions of xdist and execnet we should be running on the main thread

I realized that for xdist and execnet we should strongly recommend spawn instead of fork

[diranged]

With the latest versions of xdist and execnet we should be running on the main thread

I realized that for xdist and execnet we should strongly recommend spawn instead of fork

How do we configure spawn vs fork?

[RonnyPfannschmidt]

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

[RonnyPfannschmidt]

we learned that theres no god reason for execnet to use the lowlevel primitives - https://github.com/pytest-dev/execnet/pull/336 is a experiment to use the high level ones again

[diranged]

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Sorry I misunderstood - I got the impression there was some way in pytest-xdist to configure how it handled creating new threads...