Code coverage measurement starts too late?

[thedrow]

As evident by https://codecov.io/gh/celery/celery/src/master/celery/utils/threads.py#L58 it seems that the code is loaded before coverage measurement starts which skews our score. The coverage.py FAQ says:

Q: Why do the bodies of functions (or classes) show as executed, but the def lines do not?

This happens because coverage.py is started after the functions are defined. The definition lines are executed without coverage measurement, then coverage.py is started, then the function is called. This means the body is measured, but the definition of the function itself is not.

To fix this, start coverage.py earlier. If you use the command line to run your program with coverage.py, then your entire program will be monitored. If you are using the API, you need to call coverage.start() before importing the modules that define your functions.

I'm not sure how to fix this or if it is fixable. Any guidance is appreciated.

[ionelmc]

Your problem is in celery/contrib/pytest.py - it loads as a pytest plugin, therefore eluding all start hooks in pytest-cov.

There is a workaround you can do: force the pytest-cov engine to start early (before pytest). I made a branch here https://github.com/celery/celery/tree/preload-pytest-cov - lets see if it works.

[frankV]

Is there a general way to accomplish this. I tried adding the second pytest call with --cov-append but it still returns the same coverage report. I'm testing a flask application and this is exactly what's happening to our coverage report.

If it helps with advice, the code is here: https://github.com/boulder-python/boulderpython.org/tree/develop

Update: I know what's happening. I'm using flask's cli and it makes an instance of my flask application before it calls pytest.main. I thought it would be handy to have this custom command but I think we're gonna have to stick with good ol' py.test!

[mark0978]

I'm using pytest -n 8 -xv --cov --cov-config ./.coveragerc --cov-report html --cov-append -s and pytest -n 1 -xv --cov --cov-config ./.coveragerc --cov-report html --cov-append -s and it appears coverage is getting started after some of the modules have been imported.

Running coverage run which pytest solves this problem, but it seems like the pytest-cov plugin shouldn't need this.

The bizarre part is that it doesn't affect all of the files, some of my files do have 100% coverage, but there are quite a few files where the def is not covered but the body is which is telling me something is happening in the wrong order.

I really want to use py-cov because I can run with -n 8

Any ideas? Seems similar to this problem.

[ionelmc]

@mark0978 I'll take a look at any reproducer. Anyway, I doubt the subscribers of this issue want to get notifications for similar problems, please open separate issues in the future.

[thedrow]

@ionelmc I just checked your branch and the build does not submit code coverage to codecov for some reason.

It is however noticeable that your changes had no effect Your branch: https://travis-ci.org/celery/celery/jobs/336859511#L3192 Master: https://travis-ci.org/celery/celery/jobs/336859460#L3189

Both are at 84% coverage.

[ssbarnea]

I do get the same warning while running coverage for each thread with xdist, but the total coverage is the same as when running with -n 0.

Is there any change that this could be caused by imports from inside conftest.py files? If true, how can I tricky it? I tried to add a import coverage before, hoping that it could fix it but nope, no effect, and I cannot really remove the import from my conftest.py files.

[jaraco]

I notice this issue when I'm testing a project that's used by pytest (or a plugin). For example, the zipp project was reporting no coverage on the lines the execute on import. This was due to the pytest-checkdocs plugin, which relied on importlib_metadata which depends on zipp, causing the execution of those lines to happen before coverage starts.

I wonder if this issue could be addressed by removing modules from sys.modules after importing coverage but before running the tests. That would cause those modules to get executed and imported again when the tests run and thus cause the coverage to be measured properly.

[ionelmc]

I'm afraid that could be problematic and introduce side-effects, especially with code that keeps/compares references. It could be an opt-in behavior tho...

But then again, so is using coverage run or starting pytest-cov's engine via env vars.

[RonnyPfannschmidt]

I believe that we should have a better way for plugins like cov to opt into pre conftests initialization

Ideas for that have been around since the sprint in 2016, but never quite manifested