Click CliRunner & isolated file system: pytest-cov walks `.tox` folder when runner is executed twice (!)

[bittner]

Summary

We use pytest-cov driven by Tox for testing a Click CLI application. Click provides a CliRunner class for test execution and file system isolation.

When we run CliRunner().invoke(main, ["foo"]) in a test function coverage is calculated correctly, just for the source files of the project. When we make a second call in the same test function, e.g. CliRunner().invoke(main, ["bar"]) then, surprisingly, Coverage.py also walks the .tox folder, which contains the source files.

As a result, coverage drops dramatically, because double the amount of source code is considered.

NOTE

Only the combination of all of the following 3 factors makes the problem appear:

1. Run the test with pytest --cov (having pytest-cov installed)
2. Use the click CliRunner's isolated_filesystem
3. Invoke (at least) two click CLI commands using CliRunner().invoke in a single test

If you replace one of the above with an alternative solution coverage is not affected:

1. Run coverage run --source foo -m pytest, or
2. Omit the use of Click's isolated filesystem, or
3. Run only a single command in a test (that makes use of the isolated file system).

Reproducer

Versions

• Python 3.9.6
• pytest 7.0.1
• pytest-cov 3.0.0

Relevant output of tox: (partly redacted)

$ tox
GLOB sdist-make: /builds/applications/foo/setup.py
py create: /builds/applications/foo/.tox/py
py installdeps: -rrequirements.txt, cli-test-helpers, coverage[toml], pytest-cov
py inst: /builds/applications/foo/.tox/.tmp/package/1/foo-1.0.0.zip
attrs==21.4.0,beautifulsoup4==4.10.0,Brotli==1.0.9,certifi==2021.10.8,charset-normalizer==2.0.12,cli-test-helpers==2.1.0,click==8.0.3,ConfigArgParse==1.5.3,coverage==6.3.1,Flask==2.0.3,Flask-BasicAuth==0.2.0,Flask-Cors==3.0.10,gevent==21.12.0,geventhttpclient==1.5.3,greenlet==1.1.2,idna==3.3,iniconfig==1.1.1,itsdangerous==2.0.1,Jinja2==3.0.3,locust==2.8.2,MarkupSafe==2.0.1,msgpack==1.0.3,packaging==21.3,pluggy==1.0.0,psutil==5.9.0,py==1.11.0,pyparsing==3.0.7,pytest==7.0.1,pytest-cov==3.0.0,PyYAML==6.0,pyzmq==22.3.0,requests==2.27.1,roundrobin==0.0.2,six==1.16.0,soupsieve==2.3.1,tomli==2.0.1,typing_extensions==4.1.1,urllib3==1.26.8,Werkzeug==2.0.3,zope.event==4.5.0,zope.interface==5.4.0
py run-test-pre: PYTHONHASHSEED='...'
py run-test: commands[0] | pytest --cov foo --cov-report xml
============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-7.0.1, pluggy-1.0.0 -- /builds/applications/foo/.tox/py/bin/python
cachedir: .tox/py/.pytest_cache
rootdir: /builds/applications/foo, configfile: pyproject.toml
plugins: cov-3.0.0
collecting ... collected 31 items
...
----------- coverage: platform linux, python 3.9.6-final-0 -----------
Coverage XML written to file tests/coverage-report.xml
============================== 31 passed in 8.84s ==============================
py run-test: commands[1] | coverage report
Name                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------
.tox/py/lib/python3.9/site-packages/foo/__init__.py                                 3      0   100%
...
foo/__init__.py                                                                     3      0   100%
...
---------------------------------------------------------------------------------------------------
TOTAL                                                                             768    494    36%
___________________________________ summary ____________________________________
  py: commands succeeded
  congratulations :)

Config

# FILE: tox.ini
[tox]
envlist = py

[testenv]
deps =
    -r requirements.txt
    cli-test-helpers
    coverage[toml]
    pytest-cov
commands =
    pytest --cov foo --cov-report xml {posargs}
    coverage report

# FILE: pyproject.toml
[tool.coverage.xml]
output = "tests/coverage-report.xml"

[tool.pytest.ini_options]
addopts = "--junitxml=tests/unittests-report.xml --doctest-modules --color=yes --verbose"

Code

def test_bar():
    runner = CliRunner()

    with runner.isolated_filesystem():
        with open("foobar.yaml", "w") as f:
            f.write("something")

        gen_result = runner.invoke(main, ["generate"])
        assert gen_result.exit_code == 0, gen_result.output

        run_result = runner.invoke(main, ["run"])
        assert run_result.exit_code == 0, run_result.output