pytest-dev
tox tests + branch coverage + coverage combine = internal error
Hi,
I added coverage to the tests of the tox project and stumbled over a problem when trying to record branch coverage. The problem is os independent and can be reproduced easily with the code of the tox project.
versions:
coverage==4.4.1
py==1.4.34
pytest==3.2.1
pytest-cov==2.5.1
reproduce:
$ git clone [email protected]:tox-dev/tox.git
$ cd tox
$ pip install -e .
$ pip install pytest pytest-cov
$ pytest tests/test_z_cmdline.py::TestToxRun --cov-branch --cov=tox
result:
========================================================== test session starts ===========================================================
platform linux -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
tox comes from: '/home/oliver/work/tox/tox/tox/__init__.py'
rootdir: /home/oliver/work/tox/tox, inifile: tox.ini
plugins: cov-2.5.1
collected 3 items
tests/test_z_cmdline.py ...
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "[...]/_pytest/main.py", line 110, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "[...]/_pytest/main.py", line 146, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "[...]/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "[...]/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "[...]/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "[...]/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR> return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR> File "[...]/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR> wrap_controller.send(call_outcome)
INTERNALERROR> File "[...]/pytest_cov/plugin.py", line 235, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "[...]/pytest_cov/engine.py", line 154, in finish
INTERNALERROR> self.cov.stop()
INTERNALERROR> File "[...]/coverage/control.py", line 809, in combine
INTERNALERROR> self.data, aliases=aliases, data_paths=data_paths, strict=strict,
INTERNALERROR> File "[...]/coverage/data.py", line 735, in combine_parallel_data
INTERNALERROR> data.update(new_data, aliases=aliases)
INTERNALERROR> File "[...]/coverage/data.py", line 488, in update
INTERNALERROR> raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data
======================================================= 3 passed in 17.21 seconds ========================================================
There are a lot of timestamped .coverage.* files and all except for the first one contain "arc" data - the first one contains "lines" data - which causes the error.
I have not yet tried to reproduce this with vanilla coverage as the reason I use pytest-cov is that there are a lot of tests that run in a subprocess and pytest-cov takes care of that. It might be that the problem is actually in coverage.py itself but I thought I report this here first.
pytest tests/test_z_cmdline.py::TestToxRun --cov=tox works.
Ah yes, you've hit the default coverage config problem.
Coveragepy is using ".coveragerc" as a magic value that means: use .coveragerc if it exists, otherwise lookup a bunch of other files for configuration. So we can't just make it an absolute path (that failing test is changing CWD btw)
I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?
Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:
pytest --cov-config=tox.ini --cov=tox
So that works and is good enough for me at the moment.
I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?
Actually I'm not really sure what I meant, I think I wrote that while drunk. Can we close this?
Well the cause is mismatched coverage configuration when subprocesses are used (eg: one process has branch coverage enabled while the other doesn't). If you have a reproducer I'd gladly* look at it.
* if it's minimal :D
Hi, we have this problem with molPX. Unfortunately it has a lot of dependencies and uses conda-build for building/testing. But we are running into exact the same problem here. We have a .coveragerc file with branch coverage enabled, as well as parallel=True, because we use sub-processes (eg. jupyter kernels) to execute tests. This is performed via the pytest plugin nbval. The measurements are done by pytest-cov. We are coping the .coveragerc file to the testing directory, so it should be in place. We then also tried to pass it directly as cov-config value, but it seems to be ignored at least once.
If you're willing to take a look, the stuff is here: https://github.com/markovmodel/molPX/tree/test_
You can use export COVERAGE_DEBUG=process,config,dataio (or similar, see coverage.py docs) to debug this (look at "config_files", cwds etc).
Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:
pytest --cov-config=tox.ini --cov=toxSo that works and is good enough for me at the moment.
Do we still need the workaround of passing explicitly --cov-config=pyproject.toml now that pytest is defining the root dir relative to pyproject.toml as of 8.1.0? I am referring to: https://github.com/pytest-dev/pytest/pull/11962
