no coverage.xml when using via tox

[nschloe]

I usually run pytest-cov via tox, with tox.ini being

[tox]
envlist = py3
isolated_build = True

[testenv]
    pytest --cov foobar

With this, tox does not create a coverage.xml. It all works when running

pytest --cov foobar

directly from the command line.

This is fixable via

[testenv]
    pytest --cov foobar --cov-report xml --cov-report term

but I'm wondering if such verbosity is necessary.

[ionelmc]

You probably want to use addopts: https://docs.pytest.org/en/latest/reference.html?highlight=addopts#confval-addopts

We'd have people asking to remove it if we'd enable reporting by default.

[nschloe]

Thanks for the reply!

Well, the weird thing is that I do get the reports when calling pytest, but not via tox. What's the reason for this?

[ionelmc]

Hard to say without seeing your tox.ini/pytest.ini

[nschloe]

Right. I don't have a pytest.ini, my tox.ini usually looks something like

[tox]
envlist = py3
isolated_build = True

[testenv]
deps =
    # some dependencies
    # ...
commands =
    pytest --cov {envsitepackagesdir}/packagename --cov-report xml --cov-report term

Pretty basic, right?

[ionelmc]

This seems to contradict with your initial comment:

This is fixable via

[testenv]
    pytest --cov foobar --cov-report xml --cov-report term

but I'm wondering if such verbosity is necessary.

[nschloe]

This seems to contradict with your initial comment:

How so?

[leycec]

Necrobump. We're seeing the exact same behaviour as @nschloe in beartype/beartype, where pytest-cov successfully generates coverage.xml files under pytest but not tox. pytest-cov proudly claims that it's generating coverage.xml files under tox, but it actually doesn't.

Oddly, pytest-cov does successfully generate a terminal report under both pytest and tox. Something mysterious, magical, and horrible is happening here. This saddens us, as we otherwise enjoy the company of pytest-cov.

We're currently circumventing this by manually re-running our entire friggin' test suite through pytest during our GitHub Actions-based continuous integration – but it's all a bit fragile, kludged together, and liable to fall down at the slightest provocation when something pokes it with a stick. It really would be preferable for pytest-cov to behave as expected everywhere.

At the very least, pytest-cov should explicitly check that it actually has generated a coverage.xml file when asked to do so. That would greatly assist in error reporting and general preservation of end user sanity. :exploding_head:

[ionelmc]

It is generating it, it's just that it's not placing it in the project root, because you have used changedir = {envtmpdir} and specified a relative path for the coverage.xml file ... it would go in {envtmpdir}/coverage.xml. Perhaps you should use a src layout and avoid all that nonsense? You're worrying about fragile setups after all - the adhoc project layout you have there is a fragile layout, just saying :-)

[ionelmc]

@nschloe I don't know what to make of your report anymore, can you put a reproducer somewhere?

[leycec]

It is generating it, it's just that it's not placing it in the project root, because you have used changedir = {envtmpdir}...

Fascinating! So that's it's final destination. Thanks so much for clearing up a painful ambiguity that almost replaced my Friday evening with a visit to the nearest sanatorium. ^{cue the metallica, boys!}

Unfortunately, I'm not quite sure where to go from here. We now know what went wrong. But do we know how to make that go right without abolishing changedir = {envtmpdir}? That setting is effectively mandatory for isolating tests to installed packages, which is the whole point of using tox in the first place. Right? We're certainly not disabling that.

If it's a shooting match between tox test isolation on the one hand and ptest-cov file generation on the other, tox test isolation handily wins. We can generate coverage reports without pytest-cov, but we can't isolate tests without tox.

Ideally, pytest-cov would support a command-line option named something like --cov-report-dir. Given that, we could then instruct pytest-cov to write reports to {toxinidir} rather than {envtempdir} from our tox.ini configuration: e.g.,

setenv =
    coverage: PYTEST_ADDOPTS=--cov --cov-report-dir "{toxinidir}" --cov-report term --cov-report xml {env:PYTEST_ADDOPTS:}

Of course, that option doesn't exist. Can you think of any other sane way to effect a similar outcome short of disembowelling our tox.ini configuration?

Also, this is not particularly helpful output:

Coverage XML written to file coverage.xml

That's the root of everyone's problem, really. Ideally, the above output should embed the unambiguous absolute filename rather than ambiguous basename of that file. This is what everyone wants that output to resemble instead:

Coverage XML written to file /tmp/tox-234234/coverage.xml

It's probably coverage rather than pytest-cov that emits that output, right? In that case, would it be feasible for pytest-cov to emit an additional line of output containing that absolute filename – or is that also something beyond the purview of pytest-cov?

...and specified a relative path for the coverage.xml file...

That's the default. It's a sane default. We're certainly not hardcoding absolute pathnames in our project-wide .coveragerc file.

Perhaps you should use a src layout and avoid all that nonsense?

And... we are now in subjective argumentation territory. I'm definitely not taking the bait, because we all have better things to do than uselessly yell at Internet strangers on a Saturday evening.

Suffice to say: "That's not happening and I politely disagree, but you do you."

You're worrying about fragile setups after all - the adhoc project layout you have there is a fragile layout, just saying :-)

...right. Clearly, this discussion isn't likely to yield productive fruit. Nonetheless, I'm taking the bait. That's always the wrong thing to do. Let's do it anyway!

There's nothing "adhoc" ^{...it's "ad-hoc," by the way, but whatevas} or "fragile" about the project layout leveraged by the majority of projects. What's "adhoc" and "fragile" are strongly opinionated dependencies that break the common case by enforcing subjective dogmas. But, sure; let's just shift the onus of responsibility for the lack of sufficient configurability and disambiguous output in your package onto downstream users with demeaning assumptions, because that always solves everything.

In short, we're not adding yet another layer of filesystem indirection merely because pytest-cov lacks configurability – especially because that wouldn't solve anything, because we wouldn't drop changedir = {envtmpdir} even if we did add yet another layer of filesystem indirection, which we're not. We'll just end up avoiding pytest-cov entirely, because coverage itself already seems to support the entirety of our use case: e.g.,

coverage xml -o "{toxinidir}/coverage.xml"

And... here we are. Some Saturday evenings you just can't get back. <sigh/>

[ionelmc]

That setting is effectively mandatory for isolating tests to installed packages, which is the whole point of using tox in the first place. Right?

No. :-)

In short, we're not adding yet another layer of filesystem indirection merely because pytest-cov lacks configurability

pytest-cov lacks lots of things, including time to maintain said imagined/desired options. A lot of people use pytest-cov cause it's easy to simply add a --cov option but I think that shouldn't be the selling point of this plugin. You should use it because it works correctly with pytest-xdist and will correctly include coverage from subprocesses and multiprocessing processes.

[ionelmc]

Anyway, I agree on displaying the absolute path, gonna look into applying some abspath filters on those arguments with paths.

[nschloe]

@nschloe I don't know what to make of your report anymore, can you put a reproducer somewhere?

The original question is simply: Can we have coverage reports in terminal and XML file without having to specify it via --cov-report xml --cov-report term? (Just like vanilla pytest.)