Coverage suddenly missing everywhere but unit test modules

[KelSolaar]

Describe the bug Hello,

We have had a dramatic reduction in coverage for about a month and I'm not sure what is the issue. We basically lost coverage everywhere but in the unit test modules. It is very similar to this issue: https://github.com/nedbat/coveragepy/issues/1520. It started on Github Actions where we use the Linux build to report coverage.

There is no real difference between old "successful" builds and new "broken" ones. I managed to reproduce the behaviour locally on macOs and fixed it by using sigterm = True and I can confirm that it is now fixed on CI for macOs but Windows and Linux are still not showing the expected coverage.

To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using? Python 3.9, 3.10, 3.11
2. What version of coverage.py shows the problem? The output of coverage debug sys is helpful. 6.5.0
3. What versions of what packages do you have installed? The output of pip freeze is helpful.

alabaster (0.7.13)
anyio (3.6.2)
argon2-cffi (21.3.0)
argon2-cffi-bindings (21.2.0)
arrow (1.2.3)
astor (0.8.1)
asttokens (2.2.1)
attrs (22.2.0)
babel (2.11.0)
backcall (0.2.0)
beautifulsoup4 (4.11.2)
biblib-simple (0.1.2)
black (23.1.0)
blackdoc (0.3.8)
bleach (6.0.0)
certifi (2022.12.7)
cffi (1.15.1)
cfgv (3.3.1)
charset-normalizer (3.0.1)
click (8.1.3)
comm (0.1.2)
contourpy (1.0.7)
coverage (6.5.0)
coveralls (3.3.1)
cryptography (39.0.1)
cycler (0.11.0)
debugpy (1.6.6)
decorator (5.1.1)
defusedxml (0.7.1)
distlib (0.3.6)
docopt (0.6.2)
docutils (0.17.1)
exceptiongroup (1.1.0)
execnet (1.9.0)
executing (1.2.0)
fastjsonschema (2.16.2)
filelock (3.9.0)
flynt (0.77)
fonttools (4.38.0)
fqdn (1.5.1)
identify (2.5.18)
idna (3.4)
imageio (2.25.1)
imagesize (1.4.1)
importlib-metadata (6.0.0)
iniconfig (2.0.0)
invoke (2.0.0)
ipykernel (6.21.2)
ipython (8.10.0)
ipython-genutils (0.2.0)
ipywidgets (8.0.4)
isoduration (20.11.0)
jaraco-classes (3.2.3)
jedi (0.18.2)
jeepney (0.8.0)
jinja2 (3.1.2)
jsonpointer (2.3)
jsonschema (4.17.3)
jupyter (1.0.0)
jupyter-client (8.0.3)
jupyter-console (6.5.1)
jupyter-core (5.2.0)
jupyter-events (0.6.3)
jupyter-server (2.3.0)
jupyter-server-terminals (0.4.4)
jupyterlab-pygments (0.2.2)
jupyterlab-widgets (3.0.5)
keyring (23.13.1)
kiwisolver (1.4.4)
latexcodec (2.0.1)
markdown-it-py (2.1.0)
markupsafe (2.1.2)
matplotlib (3.7.0)
matplotlib-inline (0.1.6)
mdurl (0.1.2)
mistune (2.0.5)
more-itertools (9.0.0)
mypy-extensions (1.0.0)
nbclassic (0.5.1)
nbclient (0.7.2)
nbconvert (7.2.9)
nbformat (5.7.3)
nest-asyncio (1.5.6)
networkx (2.8.8)
nodeenv (1.7.0)
notebook (6.5.2)
notebook-shim (0.2.2)
numpy (1.24.2)
packaging (23.0)
pandas (1.5.3)
pandocfilters (1.5.0)
parso (0.8.3)
pathspec (0.11.0)
pexpect (4.8.0)
pickleshare (0.7.5)
pillow (9.4.0)
pkginfo (1.9.6)
platformdirs (3.0.0)
pluggy (1.0.0)
pre-commit (3.0.4)
prometheus-client (0.16.0)
prompt-toolkit (3.0.36)
psutil (5.9.4)
ptyprocess (0.7.0)
pure-eval (0.2.2)
pybtex (0.24.0)
pybtex-docutils (1.0.2)
pycparser (2.21)
pydata-sphinx-theme (0.12.0)
pygments (2.14.0)
pygraphviz (1.10)
pyparsing (3.0.9)
pyright (1.1.294)
pyrsistent (0.19.3)
pytest (7.2.1)
pytest-cov (4.0.0)
pytest-xdist (3.2.0)
python-dateutil (2.8.2)
python-json-logger (2.0.6)
pytz (2022.7.1)
pyyaml (6.0)
pyzmq (25.0.0)
qtconsole (5.4.0)
qtpy (2.3.0)
readme-renderer (37.3)
requests (2.28.2)
requests-toolbelt (0.10.1)
restructuredtext-lint (1.4.0)
rfc3339-validator (0.1.4)
rfc3986 (2.0.0)
rfc3986-validator (0.1.1)
rich (13.3.1)
ruff (0.0.247)
scipy (1.10.0)
secretstorage (3.3.3)
send2trash (1.8.0)
six (1.16.0)
sniffio (1.3.0)
snowballstemmer (2.2.0)
soupsieve (2.4)
sphinx (4.5.0)
sphinxcontrib-applehelp (1.0.4)
sphinxcontrib-bibtex (2.5.0)
sphinxcontrib-devhelp (1.0.2)
sphinxcontrib-htmlhelp (2.0.1)
sphinxcontrib-jsmath (1.0.1)
sphinxcontrib-qthelp (1.0.3)
sphinxcontrib-serializinghtml (1.1.5)
stack-data (0.6.2)
terminado (0.17.1)
tinycss2 (1.2.1)
toml (0.10.2)
tomli (2.0.1)
tornado (6.2)
tqdm (4.64.1)
traitlets (5.9.0)
trimesh (3.20.0)
twine (4.0.2)
typing-extensions (4.5.0)
uri-template (1.2.0)
urllib3 (1.26.14)
virtualenv (20.19.0)
wcwidth (0.2.6)
webcolors (1.12)
webencodings (0.5.1)
websocket-client (1.5.1)
widgetsnbextension (4.0.5)
zipp (3.13.0)

Note that "successful" and "broken" jobs use for practical purposes the same packages.

4. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix. I have unfortunately been unable to fix the issue anywhere but macOs by using sigterm = True
5. What commands did you run?

pytest --doctest-modules --ignore=colour/examples --cov=colour colour

My .coveragerc file:

[run]
source = colour
concurrency = multiprocessing
sigterm = True
[report]
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.:
    pass

Expected behavior The coverage should have stayed the same and not changed.

Additional context None at this point but happy to try as many things as required to understand what is going on.

[nedbat]

I notice that your pip freeze output doesn't mention "coverage"? Are you sure you are using the same version? You mention 6.5.0, but perhaps it is different in the working and non-working scenarios?

I'm not sure how to help without a reproducible case.

[KelSolaar]

Hi @nedbat,

Sorry, when I copied from Github Actions, it seems like the selection from the Log Viewer missed most a chunk of the packages. I have updated the OP using the raw logs and I can confirm that they are the same in both scenarios:

• "successful"
• "broken"

The packages differences are as follows:

identify (2.5.17) --> (2.5.18)
imageio (2.25.0) --> (2.25.1)
ipykernel (6.21.1) --> (6.21.2)
jupyter-client (8.0.2) --> (8.0.3)
jupyter-console (6.5.0) --> (6.5.1)
jupyter-server (2.2.1) --> (2.3.0)
matplotlib (3.6.3) --> (3.7.0)
pyright (1.1.293) --> (1.1.294)
python-json-logger (2.0.4) --> (2.0.6)
ruff (0.0.245) --> (0.0.247)
soupsieve (2.3.2.post1) --> (2.4)
trimesh (3.19.4) --> (3.20.0)
typing-extensions (4.4.0) --> (4.5.0)

[nedbat]

Do you have any more information about how I might be able to reproduce this? Can you link us to the repo with the code?

[KelSolaar]

Hi @nedbat,

The code is here:

• Broken coverage: https://github.com/colour-science/colour/commit/682c40654f5f9951f1daddde12777d3b3a4d416d
• Successful coverage: https://github.com/colour-science/colour/commit/635e610a8ed1c5796dc1d8ff77a3e56dccde65c5

We for now only upload coverage from macOs and Python 3.11 as Windows and Linux seemed to be affected.

Cheers,

Thomas