pytest-cov
pytest-cov copied to clipboard
Published
20 hours ago •
pytest-dev
pytest-dev
Test coverage not narrowed to package source
Summary
When I run coverage directly I get a sensible report showing a coverage of my module at 83%:
$ cat .coveragerc
[run]
source = turbot
omit =
scripts/*
setup.py
$ coverage run -m pytest
================================================================================================================================================================================ test session starts ================================================================================================================================================================================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/me/source/turbot
plugins: asyncio-0.10.0, mock-3.0.0, freezegun-0.4.1, cov-2.8.1
collected 86 items
tests/test_turbot.py ......................................................................................
$ coverage report -m
Name Stmts Miss Cover Missing
--------------------------------------------------
turbot/__init__.py 661 83 87% 27-28, 84, 133, 147, 217-281, 386, 820, 879, 1080-1087, 1092-1096, 1153-1158, 1169
turbot/migrate.py 35 35 0% 3-50
--------------------------------------------------
TOTAL 696 118 83%
However when I use pytest-cov with tox I get pure madness:
$ tox
GLOB sdist-make: /Users/me/source/turbot/setup.py
clean installed: coverage==5.1,turbot==2.0.0
clean run-test-pre: PYTHONHASHSEED='818505583'
clean run-test: commands[0] | coverage erase
py37 inst-nodeps: /Users/me/source/turbot/.tox/.tmp/package/1/turbot-2.0.0.zip
py37 installed: aiohttp==3.6.2,appdirs==1.4.3,async-timeout==3.0.1,attrs==19.3.0,beautifulsoup4==4.9.0,black==19.10b0,callee==0.3.1,certifi==2020.4.5.1,chardet==3.0.4,click==7.1.1,coverage==5.1,cycler==0.10.0,discord.py==1.3.3,distlib==0.3.0,entrypoints==0.3,filelock==3.0.12,flake8==3.7.9,freezegun==0.3.15,idna==2.9,importlib-metadata==1.6.0,isort==4.3.21,kiwisolver==1.2.0,lxml==4.5.0,matplotlib==3.2.1,mccabe==0.6.1,more-itertools==8.2.0,multidict==4.7.5,numpy==1.18.2,packaging==20.3,pandas==1.0.3,pathspec==0.8.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pyflakes==2.1.1,pyparsing==2.4.7,pytest==5.4.1,pytest-asyncio==0.10.0,pytest-cov==2.8.1,pytest-freezegun==0.4.1,pytest-mock==3.0.0,python-dateutil==2.8.1,pytz==2019.3,PyYAML==5.3.1,regex==2020.4.4,requests==2.23.0,six==1.14.0,soupsieve==2.0,toml==0.10.0,tox==3.14.6,turbot==2.0.0,typed-ast==1.4.1,urllib3==1.25.9,virtualenv==20.0.18,wcwidth==0.1.9,websockets==8.1,yarl==1.4.2,zipp==3.1.0
py37 run-test-pre: PYTHONHASHSEED='818505583'
py37 run-test: commands[0] | pytest --cov=. --cov-report=term-missing --cov-report html --no-cov-on-fail
================================================================================================================================================================================ test session starts ================================================================================================================================================================================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/me/source/turbot
plugins: asyncio-0.10.0, mock-3.0.0, freezegun-0.4.1, cov-2.8.1
collected 86 items
tests/test_turbot.py ...................................................................................... [100%]
================================================================================================================================================================================= warnings summary ==================================================================================================================================================================================
.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:39: 82 tests with warnings
/Users/me/source/turbot/.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:39: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
item = pytest.Function(name, parent=collector)
.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:45: 82 tests with warnings
/Users/me/source/turbot/.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:45: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
item = pytest.Function(name, parent=collector) # To reload keywords.
-- Docs: https://docs.pytest.org/en/latest/warnings.html
---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------------------------------------------------
.tox/py37/bin/black 6 0 100%
.tox/py37/bin/flake8 6 0 100%
.tox/py37/bin/isort 6 0 100%
.tox/py37/lib/python3.7/site-packages/_black_version.py 1 0 100%
.tox/py37/lib/python3.7/site-packages/_pytest/_argcomplete.py 37 36 3% 56-106, 109
.tox/py37/lib/python3.7/site-packages/_pytest/_code/code.py 654 633 3% 1-51, 61-71, 76, 81, 83-1145, 1154-1156, 1165-1188
.tox/py37/lib/python3.7/site-packages/_pytest/_code/source.py 244 221 9% 1-36, 40-282, 295, 302-303, 310-311, 322, 325-326, 332-416
.tox/py37/lib/python3.7/site-packages/_pytest/assertion/__init__.py 79 69 13% 4-103, 113, 139-150, 157-162, 169-179
.tox/py37/lib/python3.7/site-packages/_pytest/assertion/rewrite.py 595 517 13% 2-67, 72-74, 76, 97, 111, 114, 133-134, 142-149, 154, 183, 188, 193, 196-197, 201-204, 216, 222-226, 231-334, 341-342, 349-351, 358-359, 362-364, 366-367, 371-1035, 1040-1051, 1055, 1063
.tox/py37/lib/python3.7/site-packages/_pytest/cacheprovider.py 278 232 17% 7-60, 64, 67-75, 79-106, 109, 127-229, 242-253, 255-257, 261-263, 267-272, 276-332, 337, 342-349, 355, 358-427, 429-435, 441-457, 466-467, 471-508
.tox/py37/lib/python3.7/site-packages/_pytest/capture.py 479 407 15% 5-67, 70-133, 139, 144-153, 158-187, 192-194, 197-218, 234, 243, 245, 249, 254, 259, 264-438, 442-481, 488-491, 498-513, 517, 521, 525-584, 591-601, 606, 611-627, 635-676, 680, 684-845
.tox/py37/lib/python3.7/site-packages/_pytest/compat.py 178 140 21% 4-78, 83, 96-126, 158-159, 174, 185, 189-267, 283-285, 291, 295, 306, 310, 317, 327-331, 335-417, 419
[omitting 753 lines of coverage for all my site-packages]
turbot/__init__.py 661 661 0% 3-1169
turbot/migrate.py 35 35 0% 3-50
---------------------------------------------------------------------------------------------------------------------
TOTAL 181537 128288 29%
Coverage HTML written to dir htmlcov
========================================================================================================================================================================= 86 passed, 164 warnings in 40.09s =========================================================================================================================================================================
______________________________________________________________________________________________________________________________________________________________________________________ summary ______________________________________________________________________________________________________________________________________________________________________________________
clean: commands succeeded
py37: commands succeeded
congratulations :)
Problems:
- Coverage was run across my all my site packages even though I have a
sourcedirective in the configuration file. - The report is wrong because it says that coverage for my package is 0% which is absolutely not true.
- Due to running coverage against the entire world, it takes forever to finish.
I also tried using --cov=turbot and just --cov and those fail in a totally different way. It finishes instantly and also says no coverage:
---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name Stmts Miss Cover Missing
--------------------------------------------------
turbot/__init__.py 661 661 0% 3-1169
turbot/migrate.py 35 35 0% 3-50
--------------------------------------------------
TOTAL 696 696 0%
Coverage HTML written to dir htmlcov
Expected vs actual result
I expect to see the same results as when I run coverage directly.
Reproducer
Versions
Python 3.7.6
coverage==5.1
pytest-cov==2.8.1
Config
tox.ini
[tox]
envlist = clean,py37
[testenv]
deps = -rrequirements/dev.txt
commands =
pytest --cov=turbot \
--cov-report=term-missing \
--cov-report html \
--no-cov-on-fail
[testenv:clean]
deps = coverage
skip_install = true
commands = coverage erase
Code
All code pushed to my coverage branch at https://github.com/lexicalunit/turbot/tree/coverage
Well I can't access that repository. Maybe it's private?
Regardless, please see the examples here: https://github.com/pytest-dev/pytest-cov/tree/master/examples
Sorry, I deleted the fork and am now just a collaborator on the source now. We've also moved away from requirements.txt and use poetry as well. Not sure why pytest-cov never worked for me, hopefully all the clues are in the description and there was nothing relevant lost when I deleted the fork.