pytest-benchmark
pytest-benchmark copied to clipboard
ionelmc
Prep for pull request: Cannot get code coverage to run reliably with tox
I am working on some code that I would like to contribute that implements a more compact set of X axis labels for histograms (a new format for --benchmark-name=FORMAT).
I am currently trying to get a clean run of tox against a baseline (unmodified) version of the current (3.2.3) pytest-benchmark code, and almost all the tests run cleanly with tox except on about 1/2 to 1/3 of the test environments I get errors because the coverage command fails because the .coverage file in the pytest-benchmark toplevel directory is zero length (so the coverage command fails because it cannot find the data it is expecting in the .coverage file).
At least 50% of the test environments in a tox run do not encounter this error, but a significant number of them do.
The error looks like this:
tests/test_with_testcase.py::TerribleTerribleWayToWritePatchTests::test_foo2 PASSED [ 99%]
tests/test_with_weaver.py::test_weave_fixture PASSED [ 99%]
tests/test_with_weaver.py::test_weave_method PASSED [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 1060, in execute
INTERNALERROR> return self.con.execute(sql, parameters)
INTERNALERROR> sqlite3.OperationalError: no such table: coverage_schema
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 1065, in execute
INTERNALERROR> return self.con.execute(sql, parameters)
INTERNALERROR> sqlite3.OperationalError: no such table: coverage_schema
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 272, in _read_db
INTERNALERROR> schema_version, = db.execute_one("select version from coverage_schema")
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 1093, in execute_one
INTERNALERROR> rows = list(self.execute(sql, parameters))
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 1082, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, msg))
INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/home/users/wpoxon/src/pytest-benchmark/.coverage': no such table: coverage_schema
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/_pytest/main.py", line 257, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/_pytest/main.py", line 313, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR> return meth(self, *args, **kwargs)
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR> self.cov.stop()
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/control.py", line 426, in load
INTERNALERROR> self._data.read()
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 753, in read
INTERNALERROR> with self._connect(): # TODO: doesn't look right
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 298, in _connect
INTERNALERROR> self._open_db()
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 266, in _open_db
INTERNALERROR> self._read_db()
INTERNALERROR> File "/home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/lib/python3.8/site-packages/coverage/sqldata.py", line 274, in _read_db
INTERNALERROR> raise CoverageException(
INTERNALERROR> coverage.misc.CoverageException: Data file '/home/users/wpoxon/src/pytest-benchmark/.coverage' doesn't seem to be a coverage data file: Couldn't use data file '/home/users/wpoxon/src/pytest-benchmark/.coverage': no such table: coverage_schema
======================================================================== 1 failed, 218 passed, 12 skipped, 1 warning in 424.37s (0:07:04) =========================================================================
ERROR: InvocationError for command /home/users/wpoxon/src/pytest-benchmark/.tox/py38-pytest6-pygal24-nodist-cover/bin/py.test --cov=src --cov-report=term-missing --cov-append -vv (exited with code 3)
The summary at the end of the tox run looks like this:
clean: commands succeeded
check: commands succeeded
py27-pytest46-pygal24-nodist-cover: commands succeeded
py27-pytest46-pygal24-nodist-nocov: commands succeeded
ERROR: py27-pytest46-pygal24-xdist-cover: commands failed
py27-pytest46-pygal24-xdist-nocov: commands succeeded
ERROR: pypy-pytest46-pygal24-nodist-cover: commands failed
pypy-pytest46-pygal24-nodist-nocov: commands succeeded
ERROR: pypy-pytest46-pygal24-xdist-cover: commands failed
pypy-pytest46-pygal24-xdist-nocov: commands succeeded
ERROR: py35-pytest6-pygal24-nodist-cover: commands failed
py35-pytest6-pygal24-nodist-nocov: commands succeeded
ERROR: py35-pytest6-pygal24-xdist-cover: commands failed
py35-pytest6-pygal24-xdist-nocov: commands succeeded
ERROR: py36-pytest6-pygal24-nodist-cover: commands failed
py36-pytest6-pygal24-nodist-nocov: commands succeeded
ERROR: py36-pytest6-pygal24-xdist-cover: commands failed
py36-pytest6-pygal24-xdist-nocov: commands succeeded
ERROR: py37-pytest6-pygal24-nodist-cover: commands failed
py37-pytest6-pygal24-nodist-nocov: commands succeeded
ERROR: py37-pytest6-pygal24-xdist-cover: commands failed
py37-pytest6-pygal24-xdist-nocov: commands succeeded
ERROR: py38-pytest6-pygal24-nodist-cover: commands failed
py38-pytest6-pygal24-nodist-nocov: commands succeeded
ERROR: py38-pytest6-pygal24-xdist-cover: commands failed
py38-pytest6-pygal24-xdist-nocov: commands succeeded
ERROR: pypy3-pytest6-pygal24-nodist-cover: commands failed
pypy3-pytest6-pygal24-nodist-nocov: commands succeeded
ERROR: pypy3-pytest6-pygal24-xdist-cover: commands failed
pypy3-pytest6-pygal24-xdist-nocov: commands succeeded
ERROR: report: commands failed
ERROR: docs: commands failed
I will look into the report and docs failures separately, for now I am only concerned amout the coverage failures. If there are specific log files that you would like me to attach to help diagnose the issue, please let me know.
Sorry, forgot that part...
wpoxon@dev-2:~/src/pytest-benchmark$ uname -a
Linux dev-2 5.4.0-1035-aws #37-Ubuntu SMP Wed Jan 6 21:01:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
And here are the steps to reproduce the issue I am seeing on the system above:
cd
sudo apt-get install python3-venv # venv not installed by default, we need to install it
python3 -m venv .toxbase
.toxbase/bin/pip list # not much, but we have have pip!
.toxbase/bin/pip install tox
.toxbase/bin/tox --version
mkdir ~/bin # edit your $PATH in ~/.profile or ~/.bash_profile to include ${HOME}/bin
ln -s ~/.toxbase/bin/tox ~/bin/tox
which tox # tells me ${HOME}/bin/tox
tox --version # works
PY_VERSIONS="2.7.17 3.5.10 3.6.12 3.7.9 3.8.6 pypy-5.7.1 pypy3.6-7.3.1"
for ver in ${PY_VERSIONS}
do
pyenv uninstall -f ${ver}
pyenv install -f ${ver}
done
pyenv global ${PY_VERSIONS}
pyenv versions
cd ~src/pytest-benchmark
rm -rf .tox
tox
What appears to be happening is that sometime after the first successful execution of tests in an environment that includes cover (py27-pytest46-pygal24-nodist-cover), the {toxinidir}/.coverage file ends up getting set to zero length.
This is causing runs in any later environments that include cover to fail (py27-pytest46-pygal24-xdist-cover, pypy-pytest46-pygal24-nodist-cover, ...). I will try to narrow down when and why the {toxinidir}/.coverage file is getting set to zero length.
To help track this down, I created a fixture that runs code before and after each test. The fixture checks if {toxinidir}/.coverage exists. If the file exists and its size is 0, the fixture will assert.
I structured the fixture this way because If {toxinidir}/.coverage does not exist, there is no problem. If {toxinidir}/.coverage exists and has valid content, there is no problem. If {toxinidir}/.coverage exists and has invalid content (when I see this it is because the file is empty / size == 0), then subsequent coverage tests will fail.
I re-ran tox with this fixture installed and narrowed the failure down to this test:
py27-pytest46-pygal24-xdist-cover start: run-test
py27-pytest46-pygal24-xdist-cover run-test: commands[0] | py.test --cov=src --cov-report=term-missing --cov-append -vv
setting PATH=/home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin:/home/users/wpoxon/.pyenv/plugins/pyenv-virtualenv/shims:/home/users/wpoxon/.pyenv/shims:/home/users/wpoxon/.pyenv/bin:/home/users/wpoxon/.pyenv/bin:/home/users/wpoxon/.local/bin:/home/users/wpoxon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[953602] /home/users/wpoxon/src/pytest-benchmark$ /home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin/py.test --cov=src --cov-report=term-missing --cov-append -vv
============================================================ test session starts ============================================================
platform linux2 -- Python 2.7.17, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 -- /home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin/python
cachedir: .tox/py27-pytest46-pygal24-xdist-cover/.pytest_cache
benchmark: 3.2.3 (defaults: timer=time.time disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/users/wpoxon/src/pytest-benchmark, inifile: setup.cfg, testpaths: tests
plugins: aspectlib-1.5.1, xdist-1.34.0, instafail-0.4.2, travis-fold-1.3.0, forked-1.3.0, cov-2.11.1, benchmark-3.2.3
collected 231 items
tests/test_benchmark.py::test_help PASSED [ 0%]
tests/test_benchmark.py::test_groups PASSED [ 0%]
tests/test_benchmark.py::test_group_by_name PASSED [ 1%]
...
tests/test_benchmark.py::test_bad_rounds PASSED [ 8%]
tests/test_benchmark.py::test_bad_rounds_2 PASSED [ 9%]
tests/test_benchmark.py::test_compare PASSED [ 9%]
tests/test_benchmark.py::test_compare ERROR [ 9%]
So what I know at this point is: When tests/test_benchmark.py::test_compare begins running within the py27-pytest46-pygal24-xdist-cover environment, {toxinidir}/.coverage exists and has a size > 0. When tests/test_benchmark.py::test_compare is done running within the py27-pytest46-pygal24-xdist-cover environment, {toxinidir}/.coverage still exists and its size is now 0 which sets up for us to begin hitting the failures reported in this issue as coverage tests in this and other environments run.
Was there ever a resolution to this issue? I'm having what I believe is the same issue, and it is quite a show stopper.
I am not aware of a resolution to this issue, but I am no longer working on the project where I encountered this, so I am afraid I cannot provide any information about whether this may have been resolved by any changes in pytest-benchmark after my last update from Jan 29, 2021.