Functions executed via multiprocessing aren't reliably being covered by coverage.py

[cacti77]

Describe the bug We have code that is run in Azure DevOps using Python's multiprocessing library executed via Python's built-in unittest module. With print() statements in the parallel code we can show that the code is consistently executed. However, coverage.py reports that these functions are covered in some pipeline runs but not others, even though no code change has happened between the runs. This means code coverage reports aren't reliable and could potentially affect our quality gate criteria for passing builds.

To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

What version of Python are you using? Python 3.8.10 on Microsoft Windows Server 2022 (Azure DevOps runner image windows-2022, with this software).

What version of coverage.py shows the problem? The output of coverage debug sys is helpful. Coverage.py, version 7.6.0 with C extension

What versions of what packages do you have installed? The output of pip freeze is helpful.

Package             Version
------------------- -----------
bokeh               3.1.1
contourpy           1.1.1
coverage            7.6.0
cycler              0.12.1
et-xmlfile          1.1.0
fonttools           4.53.1
greenlet            3.0.3
importlib_resources 6.4.0
Jinja2              3.1.4
joblib              1.4.2
kiwisolver          1.4.5
lxml                5.2.2
MarkupSafe          2.1.5
matplotlib          3.7.5
numpy               1.24.4
openpyxl            3.1.5
packaging           24.1
pandas              2.0.3
pillow              10.4.0
pip                 24.1.2
pyparsing           3.1.2
python-dateutil     2.9.0.post0
pytz                2024.1
PyYAML              6.0.1
scikit-learn        1.3.2
scipy               1.10.1
setuptools          71.1.0
six                 1.16.0
SQLAlchemy          2.0.31
threadpoolctl       3.5.0
tomli               2.0.1
tomli_w             1.0.0
tornado             6.4.1
typing_extensions   4.12.2
tzdata              2024.1
wheel               0.43.0
xyzservices         2024.6.0
zipp                3.19.2

Plus our own package we've installed (redacted).

What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix. We can't share the code unfortunately, but it was launched using Python's multiprocessing module as per here. The DevOps Windows VMs have 2 CPUs.

What commands should we run to reproduce the problem? Be specific. Include everything, even git clone, pip install, and so on. Explain like we're five! Print() statements in the function called with mp.Pool.map() show that that the function is being called consistently in parallel, but the HTML coverage report only shows that the function has coverage in some DevOps runs, but not others.

Expected behavior As the parallel code is always being executed the function should always show coverage.

Additional context Add any other context about the problem here. In the DevOps pipeline we execute numerous test modules in separate folders like this: coverage run --rcfile=.coveragerc -m unittest --failfast --verbose test_module1.py

The .coveragerc config file looks like this (with package name and omit folders obfuscated):

[run]
source_pkgs =
    our_package

omit =
    */a/b/*

branch = true

concurrency =
    multiprocessing

parallel = true

After all the tests we run coverage combine, passing it all the folder names where the tests were executed. Then run coverage report and coverage xml. We also use sed to edit coverage.xml to make the file paths more human readable. Finally we publish the edited xml file to the pipeline's artifacts using DevOps' PublishCodeCoverageResults@2 task. Code Coverage then appears in the build's results and the HTML report can be drilled down into on a per file basis.

The problem is that when we drill down into the Python module that is executed in parallel, sometimes it is shown as having been covered (green) but other times it is not (red). This is despite print() statements in the parallel code showing that the code has been executed even when coverage.py thinks it has not.