coveragepy
coveragepy copied to clipboard
Race conditions when using 'tox --parallel'
Describe the bug
I've been getting a lot of errors about "no such table" or "table already exists" when trying to run tox
with coverage reporting. It looks a fair amount like #1303, but I'm opening a new issue because it looks like at least one problem was already solved in that ticket, and the existing issue has gotten kind of complicated over time.
This means that users can't use parallel testing at the same time as coverage reporting.
To Reproduce
- Check out the minimal project I created to show the bug:
git clone [email protected]:kenahoo/pkg-test-coverage-bug.git
- Have a working version of python 3.8 and 3.9 available to
tox
. - Run
python -m tox
and (hopefully) observe that everything passes. - Run
python -m tox --parallel auto
and (hopefully) observe that things fail with the above errors.
If the above doesn't demonstrate the problem, I can try to provide additional context, let me know.
Notes This is important in my organization, because we need to assure compatibility with several dependency chains, and the tests take a while, so parallel testing is very important to get timely feedback.
I don't see the errors you mention:
% python -m tox --parallel auto
⠙ [6/6] py38-more_itertools812 | py38-more_itertools813 | py38-more_itertools814 | py39-more_itertools812 |.../usr/local/virtualenvs/tmp-46d8ab1a6519ed3/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
py39-more_itertools814: OK ✔ in 3.93 seconds
py39-more_itertools812: OK ✔ in 3.96 seconds
py38-more_itertools813: OK ✔ in 3.96 seconds
py38-more_itertools814: OK ✔ in 3.99 seconds
py39-more_itertools813: OK ✔ in 3.99 seconds
py38-more_itertools812: OK (4.05=setup[3.03]+cmd[0.60,0.21,0.21] seconds)
py38-more_itertools813: OK (3.96=setup[2.92]+cmd[0.61,0.22,0.22] seconds)
py38-more_itertools814: OK (3.99=setup[2.96]+cmd[0.60,0.21,0.21] seconds)
py39-more_itertools812: OK (3.96=setup[2.93]+cmd[0.57,0.22,0.23] seconds)
py39-more_itertools813: OK (3.99=setup[2.97]+cmd[0.57,0.22,0.22] seconds)
py39-more_itertools814: OK (3.93=setup[2.90]+cmd[0.58,0.22,0.23] seconds)
congratulations :) (4.30 seconds)
But I wouldn't expect one .coverage file to be used by multiple processes simultaneously anyway. You can use the --parallel
(or [run] parallel=true
setting) to have a different data file for each process, then use coverage combine
to combine them together before reporting.
I'll close this issue, but please do re-open if I've misunderstood something.
Hmm, I'm curious why we're seeing different results. I just did a fresh clone of the repo and ran the parallel instantiation, here's the output:
% python -m tox --parallel auto
py38-more_itertools814: FAIL ✖ in 1.68 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/7/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item
test/test_code.py . [100%]
======================================================================================= 1 passed in 0.01s ========================================================================================
py38-more_itertools814: exit 1 (0.30 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31356
py38-more_itertools812: FAIL ✖ in 1.69 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/12/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item
test/test_code.py . [100%]
======================================================================================= 1 passed in 0.02s ========================================================================================
py38-more_itertools812: exit 1 (0.31 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31354
py39-more_itertools812: OK ✔ in 1.77 seconds
py39-more_itertools814: OK ✔ in 1.81 seconds
py39-more_itertools813: OK ✔ in 1.83 seconds
py38-more_itertools812: FAIL code 1 (1.69=setup[1.38]+cmd[0.31] seconds)
py38-more_itertools813: OK (1.88=setup[1.40]+cmd[0.30,0.09,0.08] seconds)
py38-more_itertools814: FAIL code 1 (1.68=setup[1.38]+cmd[0.30] seconds)
py39-more_itertools812: OK (1.77=setup[1.31]+cmd[0.26,0.10,0.10] seconds)
py39-more_itertools813: OK (1.83=setup[1.39]+cmd[0.26,0.10,0.09] seconds)
py39-more_itertools814: OK (1.81=setup[1.36]+cmd[0.26,0.10,0.09] seconds)
evaluation failed :( (1.93 seconds)
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
runner.run()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
exec(code, main_mod.__dict__)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
schema_version, = db.execute_one("select version from coverage_schema")
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
with self.execute(sql, parameters) as cur:
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
cur = self._execute(sql, parameters)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
sys.exit(main())
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
status = CoverageScript().command_line(argv)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
return self.do_run(options, args)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
self.coverage.save()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 733, in save
data = self.get_data()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
if self._collector and self._collector.flush_data():
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
self.covdata.add_lines(self.mapped_file_dict(self.data))
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
return method(self, *args, **kwargs)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
self._choose_lines_or_arcs(lines=True)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
with self._connect() as con:
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
self._open_db()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
self._read_db()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
self._init_db(db)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
db.executescript(SCHEMA)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
runner.run()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
exec(code, main_mod.__dict__)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
schema_version, = db.execute_one("select version from coverage_schema")
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
with self.execute(sql, parameters) as cur:
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
cur = self._execute(sql, parameters)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
sys.exit(main())
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
status = CoverageScript().command_line(argv)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
return self.do_run(options, args)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
self.coverage.save()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 733, in save
data = self.get_data()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
if self._collector and self._collector.flush_data():
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
self.covdata.add_lines(self.mapped_file_dict(self.data))
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
return method(self, *args, **kwargs)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
self._choose_lines_or_arcs(lines=True)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
with self._connect() as con:
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
self._open_db()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
self._read_db()
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
self._init_db(db)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
db.executescript(SCHEMA)
File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists
As far as your suggestion to use --parallel
, have I understood it correctly to mean applying a patch like the following to the tox.ini
?
diff --git a/tox.ini b/tox.ini
index d6910e7..c9fbab8 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,7 @@ deps = pytest
more_itertools812: more_itertools>=8.12,<8.13
more_itertools813: more_itertools>=8.13,<8.14
more_itertools814: more_itertools>=8.14,<8.15
-commands = python -m coverage run -m pytest
+commands = python -m coverage run --parallel -m pytest
+ python -m coverage combine
python -m coverage xml
python -m coverage html
After I make that change, the error changes to a different error, but it still doesn't succeed:
% python -m tox --parallel auto
py39-more_itertools813: FAIL ✖ in 1.67 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/16/test_pkg-0.1.tar.gz
py39-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item
test/test_code.py . [100%]
======================================================================================= 1 passed in 0.01s ========================================================================================
py39-more_itertools813: commands[1]> python -m coverage combine
No data to combine
py39-more_itertools813: exit 1 (0.09 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage combine pid=31541
py38-more_itertools814: FAIL ✖ in 1.7 seconds
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/13/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error
============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
result: Optional[TResult] = func()
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:738: in collect
for x in self._collectfile(path):
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
assert (
E AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools814: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31536
py38-more_itertools813: FAIL ✖ in 1.7 seconds
py38-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/17/test_pkg-0.1.tar.gz
py38-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error
============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
result: Optional[TResult] = func()
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:738: in collect
for x in self._collectfile(path):
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
assert (
E AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools813: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31537
py38-more_itertools812: FAIL ✖ in 1.7 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/15/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error
============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
result: Optional[TResult] = func()
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:738: in collect
for x in self._collectfile(path):
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
assert (
E AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools812: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31538
py39-more_itertools814: OK ✔ in 1.84 seconds
py38-more_itertools812: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
py38-more_itertools813: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
py38-more_itertools814: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
py39-more_itertools812: OK (1.88=setup[1.36]+cmd[0.26,0.10,0.08,0.08] seconds)
py39-more_itertools813: FAIL code 1 (1.67=setup[1.33]+cmd[0.25,0.09] seconds)
py39-more_itertools814: OK (1.84=setup[1.31]+cmd[0.26,0.10,0.09,0.08] seconds)
evaluation failed :( (1.94 seconds)
I'll close this issue, but please do re-open if I've misunderstood something.
@nedbat I don't seem to have the ability to re-open the issue, but I've posted the comments above.
Trying to make a reproducible environment - here's a Dockerfile
that I used, instead of relying on my local MacOS environment:
FROM ubuntu:22.04
# Install preliminaries
RUN DEBIAN_FRONTEND=noninteractive apt-get -y update && apt-get -y install git
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev
# Install pyenv
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
RUN echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
RUN echo 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
ENV HOME="/root"
ENV PYENV_ROOT $HOME/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
# Install some python versions
RUN pyenv install 3.8.16 3.9.16
RUN pyenv install 3.10.9
RUN pyenv global 3.10.9
RUN git clone https://github.com/kenahoo/pkg-test-coverage-bug.git /tmp/pkg-test-coverage-bug
WORKDIR /tmp/pkg-test-coverage-bug
RUN pyenv local 3.8.16 3.9.16
RUN pip install tox
# Now ready to run 'while true; do python -m tox --parallel=auto || break; done'
CMD /bin/bash
After that is built (using docker build -t pkg-test-coverage-bug .
), run (using docker run -it pkg-test-coverage-bug
) and run repeatedly until failure (using while true; do python -m tox --parallel=auto || break; done
).
For me, in this Docker environment, it took a few iterations to see the failure:
# !8
while true; do python -m tox --parallel=auto || break; done
py39-more_itertools812: OK ✔ in 1.73 seconds
py38-more_itertools813: OK ✔ in 2 seconds
py39-more_itertools813: OK ✔ in 2.17 seconds
py38-more_itertools814: OK ✔ in 2.21 seconds
py38-more_itertools812: OK ✔ in 2.26 seconds
py38-more_itertools812: OK (2.25=setup[1.42]+cmd[0.46,0.28,0.08] seconds)
py38-more_itertools813: OK (2.00=setup[1.17]+cmd[0.42,0.25,0.15] seconds)
py38-more_itertools814: OK (2.21=setup[1.26]+cmd[0.65,0.13,0.17] seconds)
py39-more_itertools812: OK (1.72=setup[1.04]+cmd[0.28,0.16,0.25] seconds)
py39-more_itertools813: OK (2.17=setup[1.13]+cmd[0.67,0.25,0.11] seconds)
py39-more_itertools814: OK (1.41=setup[1.05]+cmd[0.22,0.07,0.07] seconds)
congratulations :) (3.18 seconds)
py38-more_itertools812: OK ✔ in 1.59 seconds
py39-more_itertools813: OK ✔ in 1.66 seconds
py38-more_itertools813: OK ✔ in 1.85 seconds
py38-more_itertools814: OK ✔ in 2.14 seconds
py39-more_itertools812: OK ✔ in 2.21 seconds
py38-more_itertools812: OK (1.57=setup[0.97]+cmd[0.25,0.18,0.17] seconds)
py38-more_itertools813: OK (1.85=setup[1.03]+cmd[0.44,0.20,0.18] seconds)
py38-more_itertools814: OK (2.13=setup[1.20]+cmd[0.60,0.17,0.16] seconds)
py39-more_itertools812: OK (2.21=setup[1.29]+cmd[0.59,0.18,0.15] seconds)
py39-more_itertools813: OK (1.66=setup[1.16]+cmd[0.24,0.14,0.12] seconds)
py39-more_itertools814: OK (1.44=setup[1.04]+cmd[0.24,0.07,0.08] seconds)
congratulations :) (3.05 seconds)
py39-more_itertools813: OK ✔ in 1.58 seconds
py38-more_itertools814: OK ✔ in 1.86 seconds
py39-more_itertools812: OK ✔ in 1.9 seconds
py38-more_itertools813: OK ✔ in 2.1 seconds
py38-more_itertools812: OK ✔ in 2.19 seconds
py38-more_itertools812: OK (2.18=setup[1.29]+cmd[0.62,0.11,0.16] seconds)
py38-more_itertools813: OK (2.10=setup[1.22]+cmd[0.40,0.32,0.16] seconds)
py38-more_itertools814: OK (1.86=setup[1.32]+cmd[0.37,0.07,0.10] seconds)
py39-more_itertools812: OK (1.90=setup[1.02]+cmd[0.33,0.23,0.32] seconds)
py39-more_itertools813: OK (1.55=setup[1.07]+cmd[0.26,0.09,0.12] seconds)
py39-more_itertools814: OK (1.49=setup[1.11]+cmd[0.21,0.07,0.09] seconds)
congratulations :) (3.09 seconds)
py38-more_itertools813: OK ✔ in 1.51 seconds
py38-more_itertools812: OK ✔ in 1.52 seconds
py39-more_itertools813: OK ✔ in 1.51 seconds
py39-more_itertools812: OK ✔ in 1.54 seconds
py38-more_itertools814: OK ✔ in 1.58 seconds
py38-more_itertools812: OK (1.52=setup[1.13]+cmd[0.24,0.08,0.08] seconds)
py38-more_itertools813: OK (1.51=setup[1.11]+cmd[0.23,0.08,0.09] seconds)
py38-more_itertools814: OK (1.58=setup[1.19]+cmd[0.24,0.07,0.07] seconds)
py39-more_itertools812: OK (1.54=setup[1.15]+cmd[0.21,0.10,0.08] seconds)
py39-more_itertools813: OK (1.51=setup[1.15]+cmd[0.22,0.08,0.07] seconds)
py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
congratulations :) (2.52 seconds)
py39-more_itertools813: OK ✔ in 1.34 seconds
py39-more_itertools812: OK ✔ in 1.43 seconds
py38-more_itertools812: OK ✔ in 1.48 seconds
py38-more_itertools813: OK ✔ in 1.5 seconds
py38-more_itertools814: OK ✔ in 1.54 seconds
py38-more_itertools812: OK (1.48=setup[1.09]+cmd[0.25,0.07,0.07] seconds)
py38-more_itertools813: OK (1.50=setup[1.08]+cmd[0.23,0.07,0.11] seconds)
py38-more_itertools814: OK (1.54=setup[1.12]+cmd[0.27,0.07,0.07] seconds)
py39-more_itertools812: OK (1.42=setup[1.04]+cmd[0.23,0.08,0.08] seconds)
py39-more_itertools813: OK (1.34=setup[0.98]+cmd[0.21,0.07,0.08] seconds)
py39-more_itertools814: OK (1.02=setup[0.70]+cmd[0.19,0.07,0.07] seconds)
congratulations :) (2.39 seconds)
py39-more_itertools813: OK ✔ in 1.3 seconds
py39-more_itertools812: OK ✔ in 1.32 seconds
py38-more_itertools813: OK ✔ in 1.38 seconds
py38-more_itertools812: OK ✔ in 1.38 seconds
py38-more_itertools814: OK ✔ in 1.41 seconds
py38-more_itertools812: OK (1.38=setup[1.00]+cmd[0.23,0.08,0.07] seconds)
py38-more_itertools813: OK (1.38=setup[0.98]+cmd[0.24,0.08,0.07] seconds)
py38-more_itertools814: OK (1.41=setup[0.99]+cmd[0.28,0.07,0.07] seconds)
py39-more_itertools812: OK (1.32=setup[0.95]+cmd[0.20,0.08,0.08] seconds)
py39-more_itertools813: OK (1.29=setup[0.90]+cmd[0.24,0.07,0.08] seconds)
py39-more_itertools814: OK (0.99=setup[0.66]+cmd[0.19,0.07,0.07] seconds)
congratulations :) (2.33 seconds)
py39-more_itertools812: FAIL ✖ in 1.27 seconds
.pkg: _optional_hooks> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/pkg-test-coverage-bug/.tox/.tmp/package/134/test_pkg-0.1.tar.gz
py39-more_itertools812: commands[0]> python -m coverage run -m pytest
===================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools812/.pytest_cache
rootdir: /tmp/pkg-test-coverage-bug
collected 1 item
test/test_code.py . [100%]
====================================================================================================== 1 passed in 0.01s ======================================================================================================
py39-more_itertools812: commands[1]> python -m coverage xml
No data to report.
py39-more_itertools812: exit 1 (0.06 seconds) /tmp/pkg-test-coverage-bug> python -m coverage xml pid=5280
py38-more_itertools812: OK ✔ in 1.47 seconds
py38-more_itertools813: OK ✔ in 1.48 seconds
py39-more_itertools813: OK ✔ in 1.48 seconds
py38-more_itertools814: OK ✔ in 1.52 seconds
py38-more_itertools812: OK (1.47=setup[1.06]+cmd[0.24,0.10,0.07] seconds)
py38-more_itertools813: OK (1.48=setup[1.08]+cmd[0.24,0.08,0.08] seconds)
py38-more_itertools814: OK (1.52=setup[1.12]+cmd[0.25,0.07,0.07] seconds)
py39-more_itertools812: FAIL code 1 (1.27=setup[0.99]+cmd[0.22,0.06] seconds)
py39-more_itertools813: OK (1.48=setup[1.12]+cmd[0.21,0.08,0.08] seconds)
py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
evaluation failed :( (2.29 seconds)
If this is still a problem for you, can you try adding this line to your coverage configuration?
[run]
data_file = .coverage-${TOX_ENV_NAME}
Interesting, didn't think about that. So every thread will produce a different coverage report, then aggregate in the end?
It works a little better like this:
[run]
data_file = .coverage.${TOX_ENV_NAME}
(dot instead of dash)
Then each tox environment produces its own file without conflict. You can then combine them all together with:
coverage combine --data-file=.coverage
Thanks for picking up this thread again.
Here's the incantation I've been using in my tox.ini
files for a little while now:
[tox]
envlist = clean,py39-pandas{13,14,15,20},report
[testenv]
deps = pandas13: pandas>=1.3,<1.4
pandas14: pandas>=1.4,<1.5
pandas15: pandas>=1.5,<1.6
pandas20: pandas>=2.0,<2.1
-r{toxinidir}/requirements-test.txt
commands =
python -m pytest --cov=optos_common
depends =
py39-pandas{13,14,15,20}: clean
report: py39-pandas{13,14,15,20}
# Some extra magic for parallel runs - see the bottom of https://pytest-cov.readthedocs.io/en/latest/tox.html
setenv =
py39-pandas{13,14,15,20}: COVERAGE_FILE = .coverage.{envname}
[testenv:report]
deps = coverage
skip_install = true
commands =
coverage combine
coverage html
coverage xml
[testenv:clean]
# Clean up from any previous runs
deps = coverage
skip_install = true
commands = coverage erase
where requirements-text.txt
includes:
pytest~=7.2.1
coverage
pytest-cov
That's a bit different - I'd welcome your thoughts on some of the differences, including:
- using
{envname}
instead of${TOX_ENV_NAME}
- using
setenv COVERAGE_FILE
within[testenv]
, instead ofdata_file
within[run]
- using
pytest-cov
rather than rawcoverage
stuff - considered good or bad idea when usingpytest
? - anything else I'm doing that could be simplified
Just asking in the spirit of getting some broad recommendations for people in general "out there" to follow - it definitely took me a long time to get "something that works", but I'd be more than willing to scrap it for something else better if appropriate!
That's a bit different - I'd welcome your thoughts on some of the differences, including:
- using {envname} instead of ${TOX_ENV_NAME} - using setenv COVERAGE_FILE within [testenv], instead of data_file within [run] - using pytest-cov rather than raw coverage stuff - considered good or bad idea when using pytest? - anything else I'm doing that could be simplified
These all seem fine. You are using values available in the tox.ini file, I am using environment variables, but they have the same effect. I try to avoid pytest-cov, because it doesn't add things that I need, and it's one more component to understand and control.
This feels like one of those things to try to document somehow....
I'm getting something very similar to this, here is my error:
× Building wheel for django_dt_audit (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [37 lines of output]
....
running install_scripts
error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/django_dt_audit-0.41-py3.10.egg-info'
[end of output]
I have environments 6 environments, and run 4-sh at a time, one python version at a time.
py3{10}-django22
py3{10}-django32
py3{10,11,12}-django40
py3{10,11,12}-django41
py3{10,11,12}-django42
py3{10,11,12}-django50
and one of the build always fails randomly. Can't figure out why.