pytest-cov
pytest-cov copied to clipboard
pytest-dev
Exception at self.cov.stop() when invoke pytest and pytest-cov from code
Summary
I am trying to get a coverage report with definition lines included, when invoke pytest and pytest-cov from Python code directly. It is more convenience, compare with cmd invocation, to quick start a coverage check in IDE.
What I means is to do following... pytest.main(['test_script.py', '-v', '--cov-report', 'html', '--cov=pytest_cov_invoke_from_python_code'])
If pytest.main is called at same script, self.cov.stop() will throw exception. To avoid that, it need to create another script just to call the same cmd.
Expected vs actual result
start before import
The cov.start() in position 2 work, but it show "Coverage.py warning: Module pytest_cov_invoke_from_python_code was previously imported, but not measured (module-not-measured)". Of course the definition lines are excluded in that case.
start after import
When it is in position 1, it shows exception.
> self._collectors:
> <Collector at 0x1c1c16a3850: PyTracer>
> <module> : E:/NonCloud/pycharm2019_3_3_master_project_dir/gitPythonProject/self-code-project-common-utils/quick_code_reference/pytestquickref/test_script.py:18
> start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:518
> _init_for_start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:441
> __init__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\collector.py:111
> <Collector at 0x1c1c210c910: PyTracer>
> <module> : E:/NonCloud/pycharm2019_3_3_master_project_dir/gitPythonProject/self-code-project-common-utils/quick_code_reference/pytestquickref/test_script.py:26
> main : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:105
> _prepareconfig : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:257
> __call__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py:286
> _hookexec : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:93
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:84
> _multicall : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py:187
> pytest_cmdline_parse : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:836
> parse : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:1044
> _preparse : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:1001
> __call__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py:286
> _hookexec : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:93
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:84
> _multicall : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py:187
> pytest_load_initial_conftests : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\plugin.py:118
> __init__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\plugin.py:164
> start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\plugin.py:186
> start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\engine.py:190
> start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:518
> _init_for_start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:441
> __init__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\collector.py:111
> <Collector at 0x1c1c2222fd0: PyTracer>
> <module> : E:/NonCloud/pycharm2019_3_3_master_project_dir/gitPythonProject/self-code-project-common-utils/quick_code_reference/pytestquickref/test_script.py:26
> main : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\config\__init__.py:124
> __call__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py:286
> _hookexec : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:93
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:84
> _multicall : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py:187
> pytest_cmdline_main : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:240
> wrap_session : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:191
> _main : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:246
> __call__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py:286
> _hookexec : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:93
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:84
> _multicall : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py:187
> pytest_collection : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:257
> perform_collect : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:452
> _perform_collect : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:490
> genitems : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py:681
> collect_one_node : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\runner.py:382
> __call__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py:286
> _hookexec : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:93
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py:84
> _multicall : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py:187
> pytest_make_collect_report : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\runner.py:264
> from_call : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\runner.py:244
> <lambda> : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\runner.py:264
> collect : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\python.py:448
> _inject_setup_module_fixture : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\python.py:461
> obj : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\python.py:263
> _getobj : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\python.py:445
> _importtestmodule : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\python.py:513
> pyimport : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\py\_path\local.py:701
> _find_and_load : <frozen importlib._bootstrap>:991
> _find_and_load_unlocked : <frozen importlib._bootstrap>:975
> _load_unlocked : <frozen importlib._bootstrap>:671
> exec_module : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\assertion\rewrite.py:152
> <module> : E:\NonCloud\pycharm2019_3_3_master_project_dir\gitPythonProject\self-code-project-common-utils\quick_code_reference\pytestquickref\test_script.py:18
> start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:518
> _init_for_start : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py:441
> __init__ : D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\collector.py:111
>
> INTERNALERROR> Traceback (most recent call last):
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py", line 191, in wrap_session
> INTERNALERROR> session.exitstatus = doit(config, session) or 0
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\_pytest\main.py", line 247, in _main
> INTERNALERROR> config.hook.pytest_runtestloop(session=session)
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\hooks.py", line 286, in __call__
> INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
> INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
> INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pluggy\callers.py", line 203, in _multicall
> INTERNALERROR> gen.send(outcome)
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\plugin.py", line 254, in pytest_runtestloop
> INTERNALERROR> self.cov_controller.finish()
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\pytest_cov\engine.py", line 197, in finish
> INTERNALERROR> self.cov.stop()
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\control.py", line 542, in stop
> INTERNALERROR> self._collector.stop()
> INTERNALERROR> File "D:\python_vir_env\pycharm_py38_development\dev_pycpy3\lib\site-packages\coverage\collector.py", line 332, in stop
> INTERNALERROR> assert self._collectors[-1] is self, (
> INTERNALERROR> AssertionError: Expected current collector to be <Collector at 0x1c1c210c910: PyTracer>, but it's <Collector at 0x1c1c2222fd0: PyTracer>
>
> ============================== 3 passed in 0.34s ==============================
>
> Process finished with exit code 0
>
Code
test_script.py Position 1 not working, position 2 exclude definition lines.
import coverage
import pytest
cov = coverage.Coverage()
# start position 1
# cov.start()
from pytest_cov_invoke_from_python_code.worker_module import worker1, worker2, ifelse
def test_w1():
worker1()
def test_w2():
worker2()
def test_br():
ifelse(True)
if __name__ == '__main__':
# start position 2
cov.start()
pytest.main(['test_script.py', '-v', '--cov-report', 'html', '--cov=pytest_cov_invoke_from_python_code'])
swapper_script.py To get a "correct" coverage report, it need to call from another script.
if __name__ == '__main__':
pytest.main(['test_script.py', '-v', '--cov-report', 'html', '--cov=pytest_cov_invoke_from_python_code'])
I don't get where pytest.main is called two times in your example script.
Either way, you should not be using pytest-cov if you also use coverage directly. Pick one.
Do you means it shoud call cov.start(['test_script.py', '-v', '--cov-report', 'html', '--cov=pytest_cov_invoke_from_python_code']) instead?
I am sorry, since I can't find the relevance doc about how to use cov.start()... Also most code snippet I found online use cov = coverage.Coverage() cov.start()
Btw the test_script.py, call pytest.main(['test_script.py' ... and swapper_script.py call pytest.main(['test_script.py', ... So only call once for each script, right?
Ooook, lets get some basics in first. There are two distinct projects involved here:
-
https://pypi.org/project/coverage/ - this implements a tracer and various storage and reporting capabilities. This projects gives you that
cov = coverage.Coverage()api. -
https://pypi.org/project/pytest-cov/ - this implements a pytest plugin that, to simplify, does
cov = coverage.Coverage(); cov.start()for you while making sure all goes well when suprocesses or pytest-xdist is used.
Now I hope you can imagine how doing cov = coverage.Coverage(); cov.start() two times would be problematic, as python can only have 1 tracer active.
Explain why you need to use coverage directly, as it seems unnecessary to me (pytest-cov already does that for you, the right way).
oh...I got your point!
So let me explain more below.
The test_script.py attach above is a self contain test script. It is a test script in pytest and it can start "itself". It is very much convience for development(as a Pythoneer, we all like fast development, isn't it?)
However, that kind of self contain test script don't produce a completed coverage report, the definitions are excluded.
To get a right coverage report, it need to start it from cmd line or write an other script like swapper_script.py to initial the pytest.
Either way is not convenience...I think it is fascinating that just click F9-run to get everything instead of open up a console or create a second script to get thing work(in my opinion :-) ).
Doesn't your IDE have pytest support?
With your single test and script file design it isn't possible to measure coverage since the code will already be imported by the time you get a change to activate coverage.
I am using pycharm community edition.
For my case, start test_script.py from test_script.py (and from pycharm run)produce coverage report that excluded definitions(but the body measure correctly).
To start test_script.py from swapper_script.py produce complete overage report.