Race conditions when using 'tox --parallel'

[kenahoo]

Describe the bug I've been getting a lot of errors about "no such table" or "table already exists" when trying to run tox with coverage reporting. It looks a fair amount like #1303, but I'm opening a new issue because it looks like at least one problem was already solved in that ticket, and the existing issue has gotten kind of complicated over time.

This means that users can't use parallel testing at the same time as coverage reporting.

To Reproduce

1. Check out the minimal project I created to show the bug: git clone [email protected]:kenahoo/pkg-test-coverage-bug.git
2. Have a working version of python 3.8 and 3.9 available to tox.
3. Run python -m tox and (hopefully) observe that everything passes.
4. Run python -m tox --parallel auto and (hopefully) observe that things fail with the above errors.

If the above doesn't demonstrate the problem, I can try to provide additional context, let me know.

Notes This is important in my organization, because we need to assure compatibility with several dependency chains, and the tests take a while, so parallel testing is very important to get timely feedback.

[nedbat]

I don't see the errors you mention:

% python -m tox --parallel auto
⠙ [6/6] py38-more_itertools812 | py38-more_itertools813 | py38-more_itertools814 | py39-more_itertools812 |.../usr/local/virtualenvs/tmp-46d8ab1a6519ed3/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
py39-more_itertools814: OK ✔ in 3.93 seconds
py39-more_itertools812: OK ✔ in 3.96 seconds
py38-more_itertools813: OK ✔ in 3.96 seconds
py38-more_itertools814: OK ✔ in 3.99 seconds
py39-more_itertools813: OK ✔ in 3.99 seconds
  py38-more_itertools812: OK (4.05=setup[3.03]+cmd[0.60,0.21,0.21] seconds)
  py38-more_itertools813: OK (3.96=setup[2.92]+cmd[0.61,0.22,0.22] seconds)
  py38-more_itertools814: OK (3.99=setup[2.96]+cmd[0.60,0.21,0.21] seconds)
  py39-more_itertools812: OK (3.96=setup[2.93]+cmd[0.57,0.22,0.23] seconds)
  py39-more_itertools813: OK (3.99=setup[2.97]+cmd[0.57,0.22,0.22] seconds)
  py39-more_itertools814: OK (3.93=setup[2.90]+cmd[0.58,0.22,0.23] seconds)
  congratulations :) (4.30 seconds)

But I wouldn't expect one .coverage file to be used by multiple processes simultaneously anyway. You can use the --parallel (or [run] parallel=true setting) to have a different data file for each process, then use coverage combine to combine them together before reporting.

I'll close this issue, but please do re-open if I've misunderstood something.

[kenahoo]

Hmm, I'm curious why we're seeing different results. I just did a fresh clone of the repo and ran the parallel instantiation, here's the output:

% python -m tox --parallel auto
py38-more_itertools814: FAIL ✖ in 1.68 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/7/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.01s ========================================================================================
py38-more_itertools814: exit 1 (0.30 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31356
py38-more_itertools812: FAIL ✖ in 1.69 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/12/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.02s ========================================================================================
py38-more_itertools812: exit 1 (0.31 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run -m pytest pid=31354
py39-more_itertools812: OK ✔ in 1.77 seconds
py39-more_itertools814: OK ✔ in 1.81 seconds
py39-more_itertools813: OK ✔ in 1.83 seconds
  py38-more_itertools812: FAIL code 1 (1.69=setup[1.38]+cmd[0.31] seconds)
  py38-more_itertools813: OK (1.88=setup[1.40]+cmd[0.30,0.09,0.08] seconds)
  py38-more_itertools814: FAIL code 1 (1.68=setup[1.38]+cmd[0.30] seconds)
  py39-more_itertools812: OK (1.77=setup[1.31]+cmd[0.26,0.10,0.10] seconds)
  py39-more_itertools813: OK (1.83=setup[1.39]+cmd[0.26,0.10,0.09] seconds)
  py39-more_itertools814: OK (1.81=setup[1.36]+cmd[0.26,0.10,0.09] seconds)
  evaluation failed :( (1.93 seconds)
Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
    runner.run()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
    exec(code, main_mod.__dict__)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
    schema_version, = db.execute_one("select version from coverage_schema")
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
    with self.execute(sql, parameters) as cur:
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
    cur = self._execute(sql, parameters)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
    status = CoverageScript().command_line(argv)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
    return self.do_run(options, args)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
    self.coverage.save()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 733, in save
    data = self.get_data()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
    if self._collector and self._collector.flush_data():
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
    return method(self, *args, **kwargs)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
    self._open_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
    self._read_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
    self._init_db(db)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
    db.executescript(SCHEMA)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools814/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
    self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists
Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 837, in do_run
    runner.run()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/execfile.py", line 198, in run
    exec(code, main_mod.__dict__)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
SystemExit: ExitCode.OK

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1121, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1126, in _execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
    schema_version, = db.execute_one("select version from coverage_schema")
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1178, in execute_one
    with self.execute(sql, parameters) as cur:
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1151, in execute
    cur = self._execute(sql, parameters)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1143, in _execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/private/tmp/pkg-test-coverage-bug/.coverage': no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/kwilliams/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/__main__.py", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 950, in main
    status = CoverageScript().command_line(argv)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 665, in command_line
    return self.do_run(options, args)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/cmdline.py", line 844, in do_run
    self.coverage.save()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 733, in save
    data = self.get_data()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/control.py", line 806, in get_data
    if self._collector and self._collector.flush_data():
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/collector.py", line 475, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 239, in _wrapped
    return method(self, *args, **kwargs)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 472, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 536, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 322, in _connect
    self._open_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 267, in _open_db
    self._read_db()
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 276, in _read_db
    self._init_db(db)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 304, in _init_db
    db.executescript(SCHEMA)
  File "/private/tmp/pkg-test-coverage-bug/.tox/py38-more_itertools812/lib/python3.8/site-packages/coverage/sqldata.py", line 1226, in executescript
    self.con.executescript(script).close()
sqlite3.OperationalError: table coverage_schema already exists

As far as your suggestion to use --parallel, have I understood it correctly to mean applying a patch like the following to the tox.ini?

diff --git a/tox.ini b/tox.ini
index d6910e7..c9fbab8 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,7 @@ deps = pytest
        more_itertools812: more_itertools>=8.12,<8.13
        more_itertools813: more_itertools>=8.13,<8.14
        more_itertools814: more_itertools>=8.14,<8.15
-commands = python -m coverage run -m pytest
+commands = python -m coverage run --parallel -m pytest
+           python -m coverage combine
            python -m coverage xml
            python -m coverage html

After I make that change, the error changes to a different error, but it still doesn't succeed:

% python -m tox --parallel auto
py39-more_itertools813: FAIL ✖ in 1.67 seconds
.pkg: _optional_hooks> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /Users/kwilliams/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/16/test_pkg-0.1.tar.gz
py39-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                 

test/test_code.py .                                                                                                                                                                        [100%]

======================================================================================= 1 passed in 0.01s ========================================================================================
py39-more_itertools813: commands[1]> python -m coverage combine
No data to combine
py39-more_itertools813: exit 1 (0.09 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage combine pid=31541
py38-more_itertools814: FAIL ✖ in 1.7 seconds
py38-more_itertools814: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/13/test_pkg-0.1.tar.gz
py38-more_itertools814: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools814/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools814/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools814: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31536
py38-more_itertools813: FAIL ✖ in 1.7 seconds
py38-more_itertools813: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/17/test_pkg-0.1.tar.gz
py38-more_itertools813: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools813/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools813/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools813: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31537
py38-more_itertools812: FAIL ✖ in 1.7 seconds
py38-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /private/tmp/pkg-test-coverage-bug/.tox/.tmp/package/15/test_pkg-0.1.tar.gz
py38-more_itertools812: commands[0]> python -m coverage run --parallel -m pytest
====================================================================================== test session starts =======================================================================================
platform darwin -- Python 3.8.12, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py38-more_itertools812/.pytest_cache
rootdir: /private/tmp/pkg-test-coverage-bug
collected 0 items / 1 error                                                                                                                                                                      

============================================================================================= ERRORS =============================================================================================
_________________________________________________________________________________ ERROR collecting test session __________________________________________________________________________________
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:339: in from_call
    result: Optional[TResult] = func()
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/runner.py:370: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:738: in collect
    for x in self._collectfile(path):
.tox/py38-more_itertools812/lib/python3.8/site-packages/_pytest/main.py:572: in _collectfile
    assert (
E   AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
==================================================================================== short test summary info =====================================================================================
ERROR  - AssertionError: PosixPath('/private/tmp/pkg-test-coverage-bug/.coverage.spmacpro140.31539.834052-journal') is not a file (isdir=False, exists=False, islink=False)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 0.08s ========================================================================================
py38-more_itertools812: exit 2 (0.36 seconds) /private/tmp/pkg-test-coverage-bug> python -m coverage run --parallel -m pytest pid=31538
py39-more_itertools814: OK ✔ in 1.84 seconds
  py38-more_itertools812: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py38-more_itertools813: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py38-more_itertools814: FAIL code 2 (1.70=setup[1.34]+cmd[0.36] seconds)
  py39-more_itertools812: OK (1.88=setup[1.36]+cmd[0.26,0.10,0.08,0.08] seconds)
  py39-more_itertools813: FAIL code 1 (1.67=setup[1.33]+cmd[0.25,0.09] seconds)
  py39-more_itertools814: OK (1.84=setup[1.31]+cmd[0.26,0.10,0.09,0.08] seconds)
  evaluation failed :( (1.94 seconds)

[kenahoo]

I'll close this issue, but please do re-open if I've misunderstood something.

@nedbat I don't seem to have the ability to re-open the issue, but I've posted the comments above.

[kenahoo]

Trying to make a reproducible environment - here's a Dockerfile that I used, instead of relying on my local MacOS environment:

FROM ubuntu:22.04

# Install preliminaries
RUN DEBIAN_FRONTEND=noninteractive apt-get -y update && apt-get -y install git
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev


# Install pyenv
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
RUN echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
RUN echo 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
ENV HOME="/root"
ENV PYENV_ROOT $HOME/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH

# Install some python versions
RUN pyenv install 3.8.16 3.9.16
RUN pyenv install 3.10.9
RUN pyenv global 3.10.9

RUN git clone https://github.com/kenahoo/pkg-test-coverage-bug.git /tmp/pkg-test-coverage-bug
WORKDIR /tmp/pkg-test-coverage-bug
RUN pyenv local 3.8.16 3.9.16
RUN pip install tox

# Now ready to run 'while true; do python -m tox --parallel=auto || break; done'


CMD /bin/bash

After that is built (using docker build -t pkg-test-coverage-bug .), run (using docker run -it pkg-test-coverage-bug) and run repeatedly until failure (using while true; do python -m tox --parallel=auto || break; done).

For me, in this Docker environment, it took a few iterations to see the failure:

# !8
while true; do python -m tox --parallel=auto || break; done
py39-more_itertools812: OK ✔ in 1.73 seconds
py38-more_itertools813: OK ✔ in 2 seconds
py39-more_itertools813: OK ✔ in 2.17 seconds
py38-more_itertools814: OK ✔ in 2.21 seconds
py38-more_itertools812: OK ✔ in 2.26 seconds
  py38-more_itertools812: OK (2.25=setup[1.42]+cmd[0.46,0.28,0.08] seconds)
  py38-more_itertools813: OK (2.00=setup[1.17]+cmd[0.42,0.25,0.15] seconds)
  py38-more_itertools814: OK (2.21=setup[1.26]+cmd[0.65,0.13,0.17] seconds)
  py39-more_itertools812: OK (1.72=setup[1.04]+cmd[0.28,0.16,0.25] seconds)
  py39-more_itertools813: OK (2.17=setup[1.13]+cmd[0.67,0.25,0.11] seconds)
  py39-more_itertools814: OK (1.41=setup[1.05]+cmd[0.22,0.07,0.07] seconds)
  congratulations :) (3.18 seconds)
py38-more_itertools812: OK ✔ in 1.59 seconds
py39-more_itertools813: OK ✔ in 1.66 seconds
py38-more_itertools813: OK ✔ in 1.85 seconds
py38-more_itertools814: OK ✔ in 2.14 seconds
py39-more_itertools812: OK ✔ in 2.21 seconds
  py38-more_itertools812: OK (1.57=setup[0.97]+cmd[0.25,0.18,0.17] seconds)
  py38-more_itertools813: OK (1.85=setup[1.03]+cmd[0.44,0.20,0.18] seconds)
  py38-more_itertools814: OK (2.13=setup[1.20]+cmd[0.60,0.17,0.16] seconds)
  py39-more_itertools812: OK (2.21=setup[1.29]+cmd[0.59,0.18,0.15] seconds)
  py39-more_itertools813: OK (1.66=setup[1.16]+cmd[0.24,0.14,0.12] seconds)
  py39-more_itertools814: OK (1.44=setup[1.04]+cmd[0.24,0.07,0.08] seconds)
  congratulations :) (3.05 seconds)
py39-more_itertools813: OK ✔ in 1.58 seconds
py38-more_itertools814: OK ✔ in 1.86 seconds
py39-more_itertools812: OK ✔ in 1.9 seconds
py38-more_itertools813: OK ✔ in 2.1 seconds
py38-more_itertools812: OK ✔ in 2.19 seconds
  py38-more_itertools812: OK (2.18=setup[1.29]+cmd[0.62,0.11,0.16] seconds)
  py38-more_itertools813: OK (2.10=setup[1.22]+cmd[0.40,0.32,0.16] seconds)
  py38-more_itertools814: OK (1.86=setup[1.32]+cmd[0.37,0.07,0.10] seconds)
  py39-more_itertools812: OK (1.90=setup[1.02]+cmd[0.33,0.23,0.32] seconds)
  py39-more_itertools813: OK (1.55=setup[1.07]+cmd[0.26,0.09,0.12] seconds)
  py39-more_itertools814: OK (1.49=setup[1.11]+cmd[0.21,0.07,0.09] seconds)
  congratulations :) (3.09 seconds)
py38-more_itertools813: OK ✔ in 1.51 seconds
py38-more_itertools812: OK ✔ in 1.52 seconds
py39-more_itertools813: OK ✔ in 1.51 seconds
py39-more_itertools812: OK ✔ in 1.54 seconds
py38-more_itertools814: OK ✔ in 1.58 seconds
  py38-more_itertools812: OK (1.52=setup[1.13]+cmd[0.24,0.08,0.08] seconds)
  py38-more_itertools813: OK (1.51=setup[1.11]+cmd[0.23,0.08,0.09] seconds)
  py38-more_itertools814: OK (1.58=setup[1.19]+cmd[0.24,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.54=setup[1.15]+cmd[0.21,0.10,0.08] seconds)
  py39-more_itertools813: OK (1.51=setup[1.15]+cmd[0.22,0.08,0.07] seconds)
  py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
  congratulations :) (2.52 seconds)
py39-more_itertools813: OK ✔ in 1.34 seconds
py39-more_itertools812: OK ✔ in 1.43 seconds
py38-more_itertools812: OK ✔ in 1.48 seconds
py38-more_itertools813: OK ✔ in 1.5 seconds
py38-more_itertools814: OK ✔ in 1.54 seconds
  py38-more_itertools812: OK (1.48=setup[1.09]+cmd[0.25,0.07,0.07] seconds)
  py38-more_itertools813: OK (1.50=setup[1.08]+cmd[0.23,0.07,0.11] seconds)
  py38-more_itertools814: OK (1.54=setup[1.12]+cmd[0.27,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.42=setup[1.04]+cmd[0.23,0.08,0.08] seconds)
  py39-more_itertools813: OK (1.34=setup[0.98]+cmd[0.21,0.07,0.08] seconds)
  py39-more_itertools814: OK (1.02=setup[0.70]+cmd[0.19,0.07,0.07] seconds)
  congratulations :) (2.39 seconds)
py39-more_itertools813: OK ✔ in 1.3 seconds
py39-more_itertools812: OK ✔ in 1.32 seconds
py38-more_itertools813: OK ✔ in 1.38 seconds
py38-more_itertools812: OK ✔ in 1.38 seconds
py38-more_itertools814: OK ✔ in 1.41 seconds
  py38-more_itertools812: OK (1.38=setup[1.00]+cmd[0.23,0.08,0.07] seconds)
  py38-more_itertools813: OK (1.38=setup[0.98]+cmd[0.24,0.08,0.07] seconds)
  py38-more_itertools814: OK (1.41=setup[0.99]+cmd[0.28,0.07,0.07] seconds)
  py39-more_itertools812: OK (1.32=setup[0.95]+cmd[0.20,0.08,0.08] seconds)
  py39-more_itertools813: OK (1.29=setup[0.90]+cmd[0.24,0.07,0.08] seconds)
  py39-more_itertools814: OK (0.99=setup[0.66]+cmd[0.19,0.07,0.07] seconds)
  congratulations :) (2.33 seconds)
py39-more_itertools812: FAIL ✖ in 1.27 seconds
.pkg: _optional_hooks> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py39-more_itertools812: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/pkg-test-coverage-bug/.tox/.tmp/package/134/test_pkg-0.1.tar.gz
py39-more_itertools812: commands[0]> python -m coverage run -m pytest
===================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/py39-more_itertools812/.pytest_cache
rootdir: /tmp/pkg-test-coverage-bug
collected 1 item                                                                                                                                                                                                              

test/test_code.py .                                                                                                                                                                                                     [100%]

====================================================================================================== 1 passed in 0.01s ======================================================================================================
py39-more_itertools812: commands[1]> python -m coverage xml
No data to report.
py39-more_itertools812: exit 1 (0.06 seconds) /tmp/pkg-test-coverage-bug> python -m coverage xml pid=5280
py38-more_itertools812: OK ✔ in 1.47 seconds
py38-more_itertools813: OK ✔ in 1.48 seconds
py39-more_itertools813: OK ✔ in 1.48 seconds
py38-more_itertools814: OK ✔ in 1.52 seconds
  py38-more_itertools812: OK (1.47=setup[1.06]+cmd[0.24,0.10,0.07] seconds)
  py38-more_itertools813: OK (1.48=setup[1.08]+cmd[0.24,0.08,0.08] seconds)
  py38-more_itertools814: OK (1.52=setup[1.12]+cmd[0.25,0.07,0.07] seconds)
  py39-more_itertools812: FAIL code 1 (1.27=setup[0.99]+cmd[0.22,0.06] seconds)
  py39-more_itertools813: OK (1.48=setup[1.12]+cmd[0.21,0.08,0.08] seconds)
  py39-more_itertools814: OK (0.97=setup[0.66]+cmd[0.18,0.06,0.07] seconds)
  evaluation failed :( (2.29 seconds)

[nedbat]

If this is still a problem for you, can you try adding this line to your coverage configuration?

[run]
data_file = .coverage-${TOX_ENV_NAME}

[hneiva]

Interesting, didn't think about that. So every thread will produce a different coverage report, then aggregate in the end?

[nedbat]

It works a little better like this:

[run]
data_file = .coverage.${TOX_ENV_NAME}

(dot instead of dash)

Then each tox environment produces its own file without conflict. You can then combine them all together with:

coverage combine --data-file=.coverage

[kenahoo]

Thanks for picking up this thread again.

Here's the incantation I've been using in my tox.ini files for a little while now:

[tox]
envlist = clean,py39-pandas{13,14,15,20},report

[testenv]
deps = pandas13: pandas>=1.3,<1.4
       pandas14: pandas>=1.4,<1.5
       pandas15: pandas>=1.5,<1.6
       pandas20: pandas>=2.0,<2.1
       -r{toxinidir}/requirements-test.txt
commands =
       python -m pytest --cov=optos_common
depends =
       py39-pandas{13,14,15,20}: clean
       report: py39-pandas{13,14,15,20}
# Some extra magic for parallel runs - see the bottom of https://pytest-cov.readthedocs.io/en/latest/tox.html
setenv =
       py39-pandas{13,14,15,20}: COVERAGE_FILE = .coverage.{envname}

[testenv:report]
deps = coverage
skip_install = true
commands =
       coverage combine
       coverage html
       coverage xml

[testenv:clean]
# Clean up from any previous runs
deps = coverage
skip_install = true
commands = coverage erase

where requirements-text.txt includes:

pytest~=7.2.1
coverage
pytest-cov

That's a bit different - I'd welcome your thoughts on some of the differences, including:

• using {envname} instead of ${TOX_ENV_NAME}
• using setenv COVERAGE_FILE within [testenv], instead of data_file within [run]
• using pytest-cov rather than raw coverage stuff - considered good or bad idea when using pytest?
• anything else I'm doing that could be simplified

Just asking in the spirit of getting some broad recommendations for people in general "out there" to follow - it definitely took me a long time to get "something that works", but I'd be more than willing to scrap it for something else better if appropriate!

[nedbat]

That's a bit different - I'd welcome your thoughts on some of the differences, including:

- using {envname} instead of ${TOX_ENV_NAME}
- using setenv COVERAGE_FILE within [testenv], instead of data_file within [run]
- using pytest-cov rather than raw coverage stuff - considered good or bad idea when using pytest?
- anything else I'm doing that could be simplified

These all seem fine. You are using values available in the tox.ini file, I am using environment variables, but they have the same effect. I try to avoid pytest-cov, because it doesn't add things that I need, and it's one more component to understand and control.

This feels like one of those things to try to document somehow....

[kingbuzzman]

I'm getting something very similar to this, here is my error:

  × Building wheel for django_dt_audit (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      ....
      running install_scripts
      error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/django_dt_audit-0.41-py3.10.egg-info'
      [end of output]

I have environments 6 environments, and run 4-sh at a time, one python version at a time.

    py3{10}-django22
    py3{10}-django32
    py3{10,11,12}-django40
    py3{10,11,12}-django41
    py3{10,11,12}-django42
    py3{10,11,12}-django50

and one of the build always fails randomly. Can't figure out why.