coveragepy
coveragepy copied to clipboard
nedbat
Test suite gets stuck when running on Windows PyPy 7.3.15
The coverage.py test suite gets stuck on Windows PyPy for PyPy version 7.3.15.
Here's an example action run that gets canceled after 6 hours. The 6 hours is GitHub Actions' timeout for jobs. The test suite itself seems to reach the stuck point after about 10 minutes.
! Run tox for pypy-3.10 5h 59m 29s
Run python -m tox -- -rfsEX
ROOT: running tox-gh
ROOT: tox-gh set pypy3
tox:install
tox:pypy3
pypy3: commands[0]> python igor.py zip_mods
pypy3: commands[1]> python setup.py --quiet build_ext --inplace
pypy3: commands[2]> python -m pip install -q -e .
pypy3: commands[3]> python igor.py test_with_core ctrace -rfsEX
Skipping tests with C tracer: core ctrace not in COVERAGE_TEST_CORES=pytrace
pypy3: commands[4]> python igor.py remove_extension
Searching for coverage\tracer.so
Searching for coverage\tracer.*.so
Searching for coverage\tracer.pyd
Searching for coverage\tracer.*.pyd
Searching for build/*/coverage\tracer.so
Searching for build/*/coverage\tracer.*.so
Searching for build/*/coverage\tracer.pyd
Searching for build/*/coverage\tracer.*.pyd
pypy3: commands[5]> python igor.py test_with_core pytrace -rfsEX
=== PyPy 3.10.13 (pypy 7.3.15.final.0) (rev fc59e61cfbff) with Python tracer (.tox\pypy3\Scripts\python.EXE) ===
bringing up nodes...
bringing up nodes...
........................................................................ [ 5%]
........................................................................ [ 10%]
........................................................................ [ 15%]
......................................................................s. [ 20%]
sssss.s.s.......sss........ss.....ssss.................................. [ 25%]
........................................................................ [ 31%]
........................................................................ [ 36%]
........................................................................ [ 41%]
........................................................................ [ 46%]
........................................................................ [ 51%]
........................................................................ [ 57%]
.................................s..........s........................... [ 62%]
........................................................................ [ 67%]
............................ss.......................................... [ 72%]
................s....................ssssssssssssssssssssss............. [ 77%]
......s......s.................s..ss.............s.s..ss................ [ 83%]
........................................................................ [ 88%]
........................................................................ [ 93%]
........................................................................ [ 98%]
Error: The operation was canceled.
The Windows PyPy 3.8 runs don't get stuck because they use PyPy 7.3.11.
The last successful PyPy 3.10 run on Windows was using PyPy 7.3.14.
The same suite doesn't hang on Mac or Ubuntu using PyPy 7.3.15.
I'm hoping it can be reproduced (and debugged) by following the contributing directions and running tox -e pypy3.
I am able to reproduce the issue, it gets stuck within 2-3 minutes on my machine.
Console output:
~\Documents\GitHub\coveragepy> pypy -m tox -e pypy3
.pkg: _optional_hooks> python "C:\Program Files\pypy3.10-v7.3.15-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
.pkg: get_requires_for_build_editable> python "C:\Program Files\pypy3.10-v7.3.15-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
.pkg: build_editable> python "C:\Program Files\pypy3.10-v7.3.15-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
pypy3: install_package> python -m pip install -U --force-reinstall --no-deps .tox\.tmp\package\2\coverage-7.4.1a0.dev1-0.editable-py3-none-any.whl
pypy3: commands[0]> python igor.py zip_mods
pypy3: commands[1]> python setup.py --quiet build_ext --inplace
pypy3: commands[2]> python -m pip install -q -e .
pypy3: commands[3]> python igor.py test_with_core ctrace
Skipping tests with C tracer: core ctrace not in COVERAGE_TEST_CORES=pytrace
pypy3: commands[4]> python igor.py remove_extension
pypy3: commands[5]> python igor.py test_with_core pytrace
=== PyPy 3.10.13 (pypy 7.3.15.final.0) (rev fc59e61cfbff) with Python tracer (.tox\pypy3\Scripts\python.EXE) ===
bringing up nodes...
bringing up nodes...
....................................................................................................................................................... [ 10%]
....................................................................s...s.....s...........................................s.....s....s....s..s..s..s.s.. [ 21%]
.s.s..s.s..s.......................................................................................................................................... [ 32%]
...........................................................ss.s........sss....s...........s........................................s................... [ 43%]
.....................
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
C:\Program Files\pypy3.10-v7.3.15-win64\Lib\threading.py:324: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
600 passed, 25 skipped in 376.38s (0:06:16)
pypy3: exit 2 (377.44 seconds) C:\Users\DanDan\Documents\GitHub\coveragepy> python igor.py test_with_core pytrace pid=22096
.pkg: _exit> python "C:\Program Files\pypy3.10-v7.3.15-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
.pkg: exit 2 (0.03 seconds) C:\Users\DanDan\Documents\GitHub\coveragepy> python "C:\Program Files\pypy3.10-v7.3.15-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta pid=1772
ROOT: [33540] KeyboardInterrupt - teardown started
pypy3: FAIL code 2 (393.69=setup[3.31]+cmd[0.56,0.56,10.50,0.83,0.48,377.44] seconds)
evaluation failed :( (394.62 seconds)
~\Documents\GitHub\coveragepy>
I'm not sure if this is relevant, but the CPU usage stays high even after ctrl+C until the terminal window is closed. Looking at the task manager, Microsoft Defender seems to be the culprit:
Is there a way to tell if Defender is running in GitHub Actions? Do we know if the same thing will happen on your machine with 7.3.14, because that version of PyPy succeeded on GitHub Actions.
I believe the Defender thing is a red herring, because it also gets stuck with Defender disabled.
Currently looking into tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[spawn], as it seems to hang when running tests directly with pytest.
@devdanzin Thanks for looking into this. I tried skipping the MultiprocessingTest's, and it still hung at the end: https://github.com/nedbat/coveragepy/actions/runs/7619979724/job/20753958961#step:5:660
Do we know if the same thing will happen on your machine with 7.3.14, because that version of PyPy succeeded on GitHub Actions.
Just tested with pypy 7.3.14 on my machine, it runs fine (no issues with Defender either):
~\Documents\GitHub\coveragepy> pypy -m tox -e pypy3
.pkg: _optional_hooks> python "C:\Program Files\pypy3.10-v7.3.14-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
.pkg: get_requires_for_build_editable> python "C:\Program Files\pypy3.10-v7.3.14-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
.pkg: build_editable> python "C:\Program Files\pypy3.10-v7.3.14-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
pypy3: install_package> python -m pip install -U --force-reinstall --no-deps .tox\.tmp\package\4\coverage-7.4.1a0.dev1-0.editable-py3-none-any.whl
pypy3: commands[0]> python igor.py zip_mods
pypy3: commands[1]> python setup.py --quiet build_ext --inplace
pypy3: commands[2]> python -m pip install -q -e .
pypy3: commands[3]> python igor.py test_with_core ctrace
Skipping tests with C tracer: core ctrace not in COVERAGE_TEST_CORES=pytrace
pypy3: commands[4]> python igor.py remove_extension
pypy3: commands[5]> python igor.py test_with_core pytrace
=== PyPy 3.10.13 (pypy 7.3.14.final.0) (rev b29ea555846562061345eb58ca9bd717e1f0a9e3) with Python tracer (.tox\pypy3\Scripts\python.EXE) ===
bringing up nodes...
bringing up nodes...
................................................................................................................ [ 7%]
..................................................................................................s............. [ 15%]
.........s.............................................s........s..s..s.s........s.ss..ss....s....s......s.s.... [ 24%]
.s.......s....................................................................................................... [ 32%]
.......s......................................................................................................... [ 40%]
................................................................................................................ [ 48%]
................................................................................................................ [ 56%]
........................................................................s...............................s........ [ 64%]
.......................................................................................s......s......ss.s..s.s... [ 72%]
.s.s..ss....s.....s..................s...s....s......s..............ss...............s.s..s.ss.ss.s..s.s......... [ 80%]
..s.............s................ss.................s............................................................ [ 89%]
................................................................................................................. [ 97%]
................................... [100%]
1331 passed, 55 skipped in 149.15s (0:02:29)
.pkg: _exit> python "C:\Program Files\pypy3.10-v7.3.14-win64\lib\site-packages\pyproject_api\_backend.py" True setuptools.build_meta
pypy3: OK (164.17=setup[3.14]+cmd[0.45,0.47,9.33,0.56,0.41,149.81] seconds)
congratulations :) (165.08 seconds)
~\Documents\GitHub\coveragepy>
I've confirmed that pytest tests/test_concurrency.py is enough to hang in 7.3.15 on my machine, but works in 7.3.14. Renaming that file to something pytest won't collect also allows a tox -e pypy3 run to pass without hangs.
I'm still betting on multiprocessing, but simply running the core code of the test doesn't trigger a hang. It seems to need both the start_method_fixture and the calls to MultiprocessingTest.make_file(), so I'm a bit lost.
I can confirm that if we completely remove test_concurrency.py from the test suite, then it all completes with no problems: https://github.com/nedbat/coveragepy/actions/runs/7626679133
My testing on Windows also sees the test suite stop here:
tests/test_concurrency.py::ConcurrencyTest::test_no_multiple_light_concurrency_in_config PASSED [ 20%]
tests/test_concurrency.py::ConcurrencyTest::test_multiprocessing_needs_config_file PASSED [ 20%]
tests/test_concurrency.py::WithoutConcurrencyModuleTest::test_missing_module[eventlet] PASSED [ 20%]
tests/test_concurrency.py::WithoutConcurrencyModuleTest::test_missing_module[gevent] PASSED [ 21%]
tests/test_concurrency.py::WithoutConcurrencyModuleTest::test_missing_module[greenlet] PASSED [ 21%]
tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[fork] SKIPPED (start_method=fork not
supported here) [ 21%]
tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[spawn]
Here's a simple way to reproduce, which gives hints about what's going wrong.
Create a .coveragerc file in a directory:
[run]
concurrency = multiprocessing
source = .
Create a code.py file in the same directory:
def work(x):
return x
import multiprocessing
import os
import sys
def process_worker_main(args):
ret = work(*args)
return os.getpid(), ret
if __name__ == "__main__": # pragma: no branch
print(sys.executable) # Prints the virtual environment PyPy in 7.3.14, but the system PyPy in 7.3.15.
multiprocessing.set_start_method("spawn")
pool = multiprocessing.Pool(3)
inputs = [(x,) for x in range(6)]
outputs = pool.imap_unordered(process_worker_main, inputs)
pool.close()
pool.join()
Run coverage from that directory with multiprocessing as concurrency:
coverage run --concurrency=multiprocessing code.py
Edit: Actually this error is the one that shows when killing coverage.exe ~Results in a never ending stream of errors that look like this~:
~\PycharmProjects\coveragepy> Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~\scoop\apps\pypy3\current\Lib\multiprocessing\spawn.py", line 102, in spawn_main
source_process = _winapi.OpenProcess(
File "~\scoop\apps\pypy3\current\Lib\_winapi.py", line 317, in OpenProcess
RaiseFromWindowsErr(0)
File "~\scoop\apps\pypy3\current\Lib\_winapi.py", line 51, in RaiseFromWindowsErr
raise_WinError(type)
File "~\scoop\apps\pypy3\current\Lib\_winapi.py", line 36, in raise_WinError
raise excep
OSError: [WinError 87] Parâmetro incorreto
The original never ending error stream contains:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~\scoop\apps\pypy3\current\Lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~\scoop\apps\pypy3\current\Lib\multiprocessing\spawn.py", line 124, in _main
preparation_data = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~\scoop\apps\pypy3\current\Lib\pickle.py", line 1777, in _load
return _Unpickler(file, fix_imports=fix_imports, buffers=buffers,
File "~\scoop\apps\pypy3\current\Lib\pickle.py", line 1226, in load
dispatch[key[0]](self)
File "~\scoop\apps\pypy3\current\Lib\pickle.py", line 1551, in load_stack_global
self.append(self.find_class(module, name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~\scoop\apps\pypy3\current\Lib\pickle.py", line 1593, in find_class
__import__(module, level=0)
ModuleNotFoundError: No module named 'coverage'
Runs to completion on 7.3.14.
Hah, installing coverage into the bare/system PyPy solves the issue. So it seems to be an artifact of running in a virtual environment that has coverage installed, but somehow picking up the system PyPy environment in multiprocessing.
If I had to guess, I'd guess it has to do with this PyPy change from this commit.
Edit: Confirmed, adding print(sys.executable) to code.py prints the system PyPy in 7.3.15, but the virtual environment PyPy in 7.3.14.
