coveragepy
coveragepy copied to clipboard
nedbat
ctest compatibility
Describe the bug When I launch C++ or Python unit tests directly, all works perfectly and coverage datas are reported. But when I launch the same tests with ctest, no datas are reported. However, sitecustomize.py is well launched and partial results are generated (setting parallel = True option), but no data is reported (when launching coverage report)..
To Reproduce How can we reproduce the problem?
- What version of Python are you using? A virtual environment based on Python 3.10.4 on Windows 10 Pro
- What version of coverage.py shows the problem? 6.4.4
(venv310tympan) c:\projects\code_tympan_build_4_prof_d>coverage debug sys
-- sys -------------------------------------------------------
coverage_version: 6.4.4
coverage_module: C:\dists\python\venv310tympan\lib\site-packages\coverage\__init__.py
tracer: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: C:\projects\code_tympan\.coveragerc
configs_read: C:\projects\code_tympan\.coveragerc
config_file: C:\projects\code_tympan\.coveragerc
config_contents: b"[run]\nsource =\n\tbin\n cython_d\n\tpython\nomit =\n 3rdparty/*\nbranch = True\n# parallel = True\ndata_file = C:\\projects\\code_tympan_build_4_prof_d\\coverage\\.coverage\ndynamic_context = test_function\n\n[report]\n# Regexes for lines to exclude from consideration\nexclude_lines =\n # Have to re-enable the standard pragma\n pragma: no cover\n\n # Don't complain about missing debug-only code:\n def __repr__\n if self\\.debug\n\n # Don't complain if tests don't hit defensive assertion code:\n raise AssertionError\n raise NotImplementedError\n\n # Don't complain if non-runnable code isn't run:\n if 0:\n if __name__ == .__main__.:\n\n # Don't complain about abstract methods, they aren't run:\n @(abc\\.)?abstractmethod\n\nignore_errors = True\n\nomit = \n bin\\*\n cython_d\\plyfile.py\n\tcython_d\\sitecustomize.py\n\n[html]\ntitle = Tympan - Couverture du code Python par les tests unitaires\ndirectory = coverage_html_report\nshow_contexts = True\n"
data_file: -none-
python: 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
platform: Windows-10-10.0.19042-SP0
implementation: CPython
executable: C:\dists\python\venv310tympan\Scripts\python.exe
def_encoding: utf-8
fs_encoding: utf-8
pid: 20300
cwd: c:\projects\code_tympan_build_4_prof_d
path: C:\dists\python\venv310tympan\Scripts\coverage.exe
C:\projects\code_tympan_build_4_prof_d\cython_d
C:\projects\code_tympan_build_4_prof_d\bin
c:\projects\code_tympan_build_4_prof_d
C:\Python310\python310.zip
C:\Python310\DLLs
C:\Python310\lib
C:\Python310
C:\dists\python\venv310tympan
C:\dists\python\venv310tympan\lib\site-packages
environment: COVERAGE_PROCESS_START = C:\projects\code_tympan\.coveragerc
COVERAGE_RCFILE = C:\projects\code_tympan\.coveragerc
COVFILE = C:\projects\code_tympan\test.cov
HOME = C:\Users\d19749
PYTHONIOENCODING = UTF8
PYTHONLEGACYWINDOWSSTDIO = 1
PYTHONPATH = C:\projects\code_tympan_build_4_prof_d\cython_d\;C:\projects\code_tympan_build_4_prof_d\bin;
PYTHON_ENV = C:\dists\python\venv310tympan
TEMP = C:\Users\d19749\AppData\Local\Temp
TMP = C:\Users\d19749\AppData\Local\Temp
TYMPAN_PYTHON_INTERP = C:\dists\python\venv310tympan\Scripts\python.exe
command_line: C:\dists\python\venv310tympan\Scripts\coverage debug sys
sqlite3_version: 2.6.0
sqlite3_sqlite_version: 3.37.2
sqlite3_temp_store: 0
sqlite3_compile_options: ATOMIC_INTRINSICS=0, COMPILER=msvc-1929, DEFAULT_AUTOVACUUM,
DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
DEFAULT_WORKER_THREADS=0, ENABLE_FTS3, ENABLE_FTS4, ENABLE_FTS5,
ENABLE_JSON1, ENABLE_MATH_FUNCTIONS, ENABLE_RTREE, MALLOC_SOFT_LIMIT=1024,
MAX_ATTACHED=10, MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500,
MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=1000, MAX_FUNCTION_ARG=127,
MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=1073741823, MAX_PAGE_SIZE=65536,
MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
MAX_VARIABLE_NUMBER=32766, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
MUTEX_W32, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1
- What versions of what packages do you have installed? The output of
pip freezeis helpful.
(venv310tympan) c:\projects\code_tympan_build_4_prof_d>pip freeze
alabaster==0.7.12
Babel==2.10.3
black==22.6.0
certifi==2022.6.15
charset-normalizer==2.1.0
click==8.1.3
colorama==0.4.5
coverage==6.4.4
cycler==0.11.0
Cython==0.29.28
descartes==1.1.0
docutils==0.18.1
flake8==4.0.1
fonttools==4.33.3
graphviz==0.20
idna==3.3
imagesize==1.3.0
Jinja2==3.0.3
kiwisolver==1.4.2
MarkupSafe==2.1.1
matplotlib==3.5.1
mccabe==0.6.1
mypy-extensions==0.4.3
numpy==1.22.3
packaging==21.3
pandas==1.4.2
pathspec==0.9.0
Pillow==9.1.0
platformdirs==2.5.2
py-spy==0.3.12
pycodestyle==2.8.0
pyflakes==2.4.0
Pygments==2.12.0
pyparsing==3.0.8
PyQt5==5.15.6
PyQt5-Qt5==5.15.2
PyQt5-sip==12.10.1
pyqtgraph==0.12.3
python-dateutil==2.8.2
pytz==2022.1
requests==2.28.1
scipy==1.8.0
Shapely==1.8.1
six==1.16.0
snowballstemmer==2.2.0
Sphinx==5.0.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
tomli==2.0.1
urllib3==1.26.9
XlsxWriter==3.0.3
-
What code shows the problem? I'm using coverage on this commit
-
What commands did you run?
coverage erase
ctest -C Debug -R test_altimetry_builder -VV
coverage combine
coverage report
This command sequence reports no data.
coverage erase
python python\tests\test_altimetry_builder.py
coverage combine
coverage report
This command sequence works perfectly and reports data.
Expected behavior Launching units test by ctest or python should have same behaviour.
Additional context
The problem occurs on Code_TYMPAN which is an opensource solution written in C++ and Python
I'm using ctest to launch all C++ and Python unit tests in one single command, ctest -C Debug --output-on-failure
Both C++ and Python unit tests can launch python processes, so I configure 'Measuring subprocesses' with sitecustomize.py option, which works fine when not using ctest.
Reproducing this behaviour can be difficult as one has to set up Code_TYMPAN development environment. If necessary, installation instructions on Windows are here
It could be easier to reproduce the behaviour on a simple CMake solution using ctest to configure and launch tests.
Thanks for your attention.
Sorry, but I don't know what ctest is, or how to use it. Can you make a simple reproducer that I can try?
