multiprocessing.resource_tracker is imported more than once

[finite-state-machine]

Describe the bug

The coverage package causes multiprocessing.resource_tracker to be imported more than once, resulting in leaked resources, warnings, and other issues.

To Reproduce

Fairly concise code to reproduce the issue is attached as a tarball, as directory structure matters. The POC depends on pip, pytest, pytest-cov, and pyftpdlib.

The files of the tarball (neglecting the directory that prevents it from being a tarbomb) are as follows:

reproduce_issue.sh

#!/usr/bin/env bash
python3 -m pip install -r requirements.txt
python3 -m pytest --pyargs --cov=some_package.some_module some_package.some_module --capture=no

requirements.txt

pyftpdlib
pytest
pytest-cov

some_package/__init__.py

from . import some_module

some_package/some_module.py

from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer

Sample output of reproduce_issue.sh (excluding pip's output):

======================================================================================================================================================================================================================================================================= test session starts =======================================================================================================================================================================================================================================================================
platform darwin -- Python 3.8.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/me/coverage_bug
plugins: cov-3.0.0
collected 0 items


---------- coverage: platform darwin, python 3.8.3-final-0 -----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
some_package/some_module.py       3      0   100%
-------------------------------------------------
TOTAL                             3      0   100%

====================================================================================================================================================================================================================================================================== no tests ran in 0.17s ======================================================================================================================================================================================================================================================================
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:221: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-1lcpmomy': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-kzffvfiw': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-fpqvsru6': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-tytkkqsh': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-wam1qd25': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py:234: UserWarning: resource_tracker: '/mp-_qu77dju': [Errno 22] Invalid argument
  warnings.warn('resource_tracker: %r: %s' % (name, e))
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-kzffvfiw'
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-fpqvsru6'
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-wam1qd25'
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-_qu77dju'
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-tytkkqsh'
rtype='semaphore'
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/resource_tracker.py", line 203, in main
    cache[rtype].remove(name)
KeyError: '/mp-1lcpmomy'

Expected behavior No errors or warnings should be issued when testing this code.

Additional context

The problem seems to happen because of coverage.misc.sys_modules_saved(), which causes the multiprocessing.resource_tracker module to be imported more than once. That module contains a singleton (_resource_tracker/ResourceTracker) responsible for tracking certain objects which can be shared across processes, including multiprocessing.Lock(), which is used by pyftpdlib.

That some_package/__init__.py imports some_package.some_module seems to be important, as does --pyargs.

It may be that excluding multiprocessing.resource_tracker from sys_modules_saved() is sufficient to solve this issue.

I apologize for any deficits in this bug report.

[nedbat]

Thanks, this is unusual. I see what you mean about sys_modules_saved, and you are right: excluding multiprocessing.resource_tracker from deletion in SysModuleSaver.restore clears up the problem.

But I'm reluctant to hard-code a list of modules to avoid re-importing, so I'm looking for other ideas.

TBH, I'm surprised this hasn't come up more. What is it in your scenario that makes this happen? I have multiprocessing tests, and they don't cause these warnings.

[nedbat]

This was the patch I tried:

diff --git a/coverage/misc.py b/coverage/misc.py
index aaf1bcf7..549d5c22 100644
--- a/coverage/misc.py
+++ b/coverage/misc.py
@@ -48,6 +48,9 @@ def isolate_module(mod):
 os = isolate_module(os)
 
 
+# Modules with singletons that don't like being re-imported.
+PERSIST_MODULES = {'multiprocessing.resource_tracker'}
+
 class SysModuleSaver:
     """Saves the contents of sys.modules, and removes new modules later."""
     def __init__(self):
@@ -57,7 +60,8 @@ def restore(self):
         """Remove any modules imported since this object started."""
         new_modules = set(sys.modules) - self.old_modules
         for m in new_modules:
-            del sys.modules[m]
+            if m not in PERSIST_MODULES:
+                del sys.modules[m]
 
 
 @contextlib.contextmanager

[finite-state-machine]

I imagine there are a couple of factors resulting in this test case being unusual. The first is that simply importing pyftpdlib results in the creation of Locks, triggering the creation of the resource tracker process. That can't be that unusual in itself.

The second factor may be that when __init__.py imports its submodule, it results in importing of that module a second time. (In my application, I have a package that is divided into multiple modules, but which exposes all of the public symbols from those modules at the package level. Hence, __init__.py imports all of its submodules.) In minimizing, I found that both --pyargs and the import statement in __init__.py were necessary to reproduce the undesired behaviour.

If there's a need, I can try to minimize or simplify the proof-of-concept further.