--cov and xdist missing coverage

[rouge8]

Running pytest-cov with xdist (py.test -n auto --cov --cov-report=html) shows a warning at the end of the test run (Coverage.py warning: No data was collected) and reports 26431/73593 covered statements versus running without xdist (py.test --cov --cov-report=html) which reports 36879/73593 covered statements.

Versions:

coverage==4.2 pytest-cov==2.3.1 pytest==2.9.2 pytest-xdist==1.15.0

.coveragerc

[run]
source = dir1,dir2,dir3,dir4
branch = True
parallel = True

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover

    # Don't complain if non-runnable code isn't run
    if __name__ == .__main__.:

    # Don't complain about uncovered code in tests that were supposed to fail
    @pytest.mark.xfail

# Always show line numbers of uncovered statements
show_missing = True

I'm going to work on a reproducible sample, but I'm hoping there's some obvious config error here.

[ionelmc]

Does this happen on an opensource project I could look at?

[ionelmc]

Also, how do you use xdist? Your example command line doesn't use xdist ...

[rouge8]

Does this happen on an opensource project I could look at?

Working on extracting something, will post a repo once I'm done.

Also, how do you use xdist? Your example command line doesn't use xdist ...

The -n auto uses xdist:

  -n numprocesses       shortcut for '--dist=load --tx=NUM*popen', you can use
                        'auto' here for auto detection CPUs number on host
                        system

[rouge8]

I can reproduce the "Coverage.py warning: No data was collected" with https://github.com/rouge8/xdist-cov/tree/225d5a0ed417420e185470c6ef40bc2b2b24d2a9, although that reports 100% coverage. I'm looking into what lines are uncovered with xdist in the actual project and will try to reproduce those too.

Also thanks for the very rapid responses! I posted this hoping it would be obvious user error, but I'll keep working at extracting a better example.

[ionelmc]

From what I see there it's a benign error (but beyond my control): the master process complains that it don't have any coverage data (rightly so, because it don't run any tests - everything is ran in a slave process when xdist is on).

Can you elaborate about the line difference in your original project? What exactly is not getting covered?

[rouge8]

Can you elaborate about the line difference in your original project? What exactly is not getting covered?

It looks like it's mostly declarative code (e.g. SQLAlchemy models), function defs, constants, and conftest.pys. Easiest way to reproduce was with a conftest.py, e.g. https://github.com/rouge8/xdist-cov/commit/25a84de20acf3ead196d69a8dfd96a99285dfdb1:

with xdist

$ tox -e py27 --  -n auto --cov --cov-report=term-missing
py27 installed: apipkg==1.4,coverage==4.2,execnet==1.4.1,py==1.4.31,pytest==2.9.2,pytest-cov==2.3.1,pytest-xdist==1.15.0
py27 runtests: PYTHONHASHSEED='2312460079'
py27 runtests: commands[0] | py.test -n auto --cov --cov-report=term-missing
================================== test session starts ===================================
platform darwin -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /Users/andy/tmp/xdist-cov, inifile: tox.ini
plugins: cov-2.3.1, xdist-1.15.0
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
..Coverage.py warning: No data was collected.


---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------
dir1/__init__.py             0      0      0      0   100%
dir1/mod1.py                 2      0      0      0   100%
dir1/tests/conftest.py       3      2      0      0    33%   1-4
dir1/tests/test_all.py       6      0      0      0   100%
--------------------------------------------------------------------
TOTAL                       11      2      0      0    82%

================================ 2 passed in 0.66 seconds ================================
________________________________________ summary _________________________________________
  py27: commands succeeded
  congratulations :)

without xdist

$ tox -e py27 -- --cov --cov-report=term-missing
py27 installed: apipkg==1.4,coverage==4.2,execnet==1.4.1,py==1.4.31,pytest==2.9.2,pytest-cov==2.3.1,pytest-xdist==1.15.0
py27 runtests: PYTHONHASHSEED='2289093109'
py27 runtests: commands[0] | py.test --cov --cov-report=term-missing
================================== test session starts ===================================
platform darwin -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /Users/andy/tmp/xdist-cov, inifile: tox.ini
plugins: cov-2.3.1, xdist-1.15.0
collected 2 items

dir1/tests/test_all.py ..

---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------
dir1/__init__.py             0      0      0      0   100%
dir1/mod1.py                 2      0      0      0   100%
dir1/tests/conftest.py       3      0      0      0   100%
dir1/tests/test_all.py       6      0      0      0   100%
--------------------------------------------------------------------
TOTAL                       11      0      0      0   100%


================================ 2 passed in 0.02 seconds ================================
________________________________________ summary _________________________________________
  py27: commands succeeded
  congratulations :)

[ionelmc]

Interesting. It turns out conftest.py is loaded before the cov plugin is started.

[rouge8]

It turns out conftest.py is loaded before the cov plugin is started.

Yeah, I suspect that's what's happening with our other files as well, since in our main conftest.py we import the database models and create the tables in a fixture, e.g.:

@pytest.yield_fixture(scope='session')
def sqlengine(_settings):
    """
    Create a database, engine, and all tables once/session.

    Drops the database at the end of the test session.
    """
    from sqlalchemy import engine_from_config
    from sqlalchemy_utils import create_database, drop_database
    from app.models import DBSession, DefaultBase

    engine = engine_from_config(_settings, prefix='sqlalchemy_master.')
    create_database(engine.url)
    DBSession.configure(bind=engine)
    try:
        DefaultBase.metadata.create_all(engine)
        yield engine
    finally:
        # Always drop the temporary test database, even if we encountered an
        # exception in creating the tables.
        drop_database(engine.url)

Importing the models will import from other files which is probably why all of the def, class, constants, etc. are uncovered.

[ionelmc]

Hmmm, we do have a test for this exact situation, and it is passing: https://github.com/pytest-dev/pytest-cov/blob/master/tests/test_pytest_cov.py#L807-L819

I wonder what's different

[rouge8]

I wonder what's different

It looks like it works when you have a separate test directory (top level app and tests), but not when your structure is app/tests... https://github.com/rouge8/xdist-cov/pull/1

[tleonhardt]

Thanks for making a great tool.

I'm seeing this problem where --cov works great, but as soon as I add -n2 to enable xdist, it misses coverage on TravisCI and AppVeyor, but works fine when I run it on my Mac.

Here is a link to an example good build on Travis (without xdist): https://travis-ci.org/python-cmd2/cmd2/jobs/249456806

And here is one to a bad build on Travis (with xdist): https://travis-ci.org/python-cmd2/cmd2/jobs/249455458

So coverage and execution time with and without exist:

• with xdist at -n2: 75.9% coverage in 83 seconds (1 min 23 sec)
• no exist: 94.3% coverage in 704 seconds (11 min 44 sec)

I see a similar performance improvement on my Mac when using xdist, but it gives me the same code coverage percentage with and without xdist.

Given the insane performance difference provided by xdist, it would be really awesome if it worked reliably with --cov.

In all cases (both my Mac and TravisCI Linux containers) the versions are:

• pytest-cov: 2.5.1
• pytest-xdist: 1.18.0

[tleonhardt]

@rouge8 Thanks for your post showing what you did to get this to work! I took a look at what you did and I made similar changes to my tox.ini and .coveragerc until things worked. I'm not sure what exactly did the trick, but something did.

@ionelmc In case it is helpful, you can see the changes I made to get it to work here: https://github.com/python-cmd2/cmd2/pull/167

[ionelmc]

@tleonhardt you can see in your report the problem:

.tox/py36/lib/python3.6/site-packages/cmd2.py    1138    909    20.1%
cmd2.py                                          1138    274    75.9%

It totals up to what you expect. The problem is that the wrong file is getting imported. You worked around the package layout problem by using tmpdir previously. I think you should switch to a src-layout (or use develop mode in tox, as a cheap workaround).

[thedrow]

Is there an actual fix to this problem?

[ionelmc]

@thedrow not sure what you're asking. Besides having a different project layout there's aliasing configuration (http://coverage.readthedocs.io/en/latest/config.html#paths). The issue of tardy coverage start I will fix sometime this month I hope.

[thedrow]

I'm asking if there is a fix that doesn't involve me changing layouts.

[ionelmc]

Sure, use develop mode (https://tox.readthedocs.io/en/latest/config.html#confval-usedevelop=BOOL)

[ionelmc]

Ok what did I mean by tardy coverage? Serious question, I forgot.

I've checked the code and we use pytest_load_initial_conftests - there's hardly any earlier hook we could use.

[manoadamro]

Got a similar problem, tests all run fine, coverage always at 0% with message "overage.py warning: No data was collected. (no-data-collected)" works fine on one of my projects and not the other.

[jxltom]

I still have this issue. Any updates?

[rynkwn]

Would also appreciate any updates!

[ionelmc]

Oooof ... I would need help with small reproducers on the latest pytest-cov release. Pretty sure people are having different problems here.

[roo-oliv]

I would like to add that I was using pytest-xdist with pytest-cov and was sometimes receiving No data to report. I would just retrigger the build and, eventually, it would pass without errors.

I decided to remove pytest-xdist from the project yesterday. Until today I thought the problem was gone but a build today failed again with the same error. Retriggered the build and it ran fine.

----------- coverage: platform linux, python 3.6.8-final-0 -----------
Name                                             Stmts   Miss  Cover
----------------------------------------------------------------------------------------
xpto/__init__.py                                    3      0   100%
xpto/my_lib/__init__.py                             0      0   100%
xpto/my_lib/cloudsearch/__init__.py                 3      0   100%
xpto/my_lib/cloudsearch/cloudsearch_client.py      21      2    90%
xpto/my_lib/cloudsearch/cloudsearch_proxy.py       34      2    94%
xpto/my_lib/errors/__init__.py                      4      0   100%
xpto/my_lib/errors/search_error.py                  3      0   100%
xpto/my_lib/errors/serializable_error.py            7      1    86%
xpto/my_lib/errors/server_error.py                  3      0   100%
xpto/my_lib/my_lib_client.py                        8      0   100%
xpto/my_lib/request_utils.py                       12      0   100%
----------------------------------------------------------------------------------------
TOTAL                                              98      5    95%
Coverage HTML written to dir unittestscov


========================== 11 passed in 1.12 seconds ===========================
No data to report.

[ionelmc]

@allrod5 does the problem still reproduce with the master of pytest-cov? pip install https://github.com/pytest-dev/pytest-cov/archive/master.zip and give it a try.

[kmuehlbauer]

I just came across this via github search. In my case I had to export COVERAGE_PROCESS_START=/path/to/.coveragerc and using the folowing lines to my .coveragerc:

[run]
concurrency = multiprocessing
parallel = true

Then xdist worked correctly together with cov.

[roo-oliv]

@ionelmc sorry for the long delay. I believe I found out what was going wrong.

Today we tested building the project with the latest release 2.7.1 (no pytest-xdist used nor installed) and the issue remained: sometimes the build failed with the message No data to report though there were coverage data printed to the terminal.

We then added pytest-xdist and used the workaround proposed by @kmuehlbauer. It worked like a charm.

Our CI system was running integration tests and unit tests at the same time and I believe that this parallel execution was messing up things for pytest-cov. pytest-xdist with the mentioned configurations may have guarded pytest-cov from race conditions.

[roo-oliv]

Following up my last message. The issue came back last week even with the proposed workarounds applied. We then experimented with reducing parallelism.

It's been a week of consistent results: this issue arouses when our CI system ran the same test suite, be it unit or integration tests, at the same time with different Python versions. The problem is not that there are different containers with different python versions running at the same time. The issue is related to running the same test suite concurrently.

[ionelmc]

@ionelmc sorry for delay, can make a small reproducer? or at least point me to a repo where the issue happens?

[xiuliren]

@ionelmc here is an example, not sure it is is exactly the same problem or not. I got only 1% coverage with all __init__.py files. I can get 45% in local run.

https://travis-ci.org/seung-lab/chunkflow/jobs/568889985

[ionelmc]

@allrod5 looks like i pinged myself instead of you. Do you have a reproducer?

@jingpengw It looks like the problem there is misconfiguration when using the adhoc layout. See https://github.com/pytest-dev/pytest-cov/tree/master/examples