pluggy
pluggy copied to clipboard
pytest-dev
Cythonize the call loop
Resolves #104 and is something I wrote a few years ago after a discussion with @RonnyPfannschmidt.
I haven't integrated the _c_multicall() into the benchmark tests yet (starting to think I shouldn't have removed the multiple loops from #147 :roll_eyes:) but this is at least a draft that builds the faster function.
Update: Above is all complete.
Interested to see what peeps think!
TODO:
- [x] fix build
- [x] revert 72948afada60cd9d6474b4631fabb864731de578
- [x] integrate with benchmark tests
- [x] make linter tests pass
- [ ] make
cythonan optional dependency? - [ ] test with pytest test suite -> deferring to @bluetech :sunglasses:
Ahh yeah so there's an import cycle in this..
I need to break out the _Result into it's own module, any preferences on a name?
I was thinking _result.py ?
It's that or we move the _c_multicall() import further down the file?
One other idea: given the .pyx code isn't yet truly written as native cython code we could just breakout _multicall() to a new module and then copy and cythonize it on build instead of having duplicate code in both files?
Might just make sense to actually play with writing properly tuned cython code to see if there's any benefit before jumping on that wagon tho.
I need to break out the _Result into it's own module, any preferences on a name? I was thinking _result.py ?
I proposed this before in #227 so :+1: from me
I proposed this before in #227 so +1 from me
@bluetech I might add it in in a new PR just so that it can come in before this one and because we're touching stuff for the 1.0 release that's importing it.
I have been working a bit on pytest performance lately, so let me know once you believe that the implementation is semantically correct, and I'll do some profiling on pytest (when I get the chance).
@bluetech sounds good!
I'm just gonna put up a PR to split out the _Result and friends and then this will basically be ready once I rebase on that.
I rebased this onto #268 so once that's merged I'll rebase onto master again to factor out those commits.
Running pytest's test suite against this branch (+ switching to _c_multicall) gives two failures:
================================================================================== FAILURES ==================================================================================
________________________________________________________________ TestTracebackCutting.test_traceback_argsetup ________________________________________________________________
self = <collect.TestTracebackCutting object at 0x7f98a766a790>, testdir = <Testdir local('/tmp/pytest-of-ran/pytest-9/test_traceback_argsetup0')>
def test_traceback_argsetup(self, testdir):
testdir.makeconftest(
"""
import pytest
@pytest.fixture
def hello(request):
raise ValueError("xyz")
"""
)
p = testdir.makepyfile("def test(hello): pass")
result = testdir.runpytest(p)
assert result.ret != 0
out = result.stdout.str()
assert "xyz" in out
assert "conftest.py:5: ValueError" in out
numentries = out.count("_ _ _") # separator for traceback entries
> assert numentries == 0
E assert 26 == 0
/home/ran/src/pytest/testing/python/collect.py:955: AssertionError
---------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7
rootdir: /tmp/pytest-of-ran/pytest-9/test_traceback_argsetup0
collected 1 item
test_traceback_argsetup.py E [100%]
==================================== ERRORS ====================================
____________________________ ERROR at setup of test ____________________________
> ???
src/pluggy/callers/cythonized.pyx:14:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
request = <SubRequest 'hello' for <Function test>>
@pytest.fixture
def hello(request):
> raise ValueError("xyz")
E ValueError: xyz
conftest.py:5: ValueError
=========================== short test summary info ============================
ERROR test_traceback_argsetup.py::test - ValueError: xyz
=============================== 1 error in 0.03s ===============================
______________________________________________________________________ test_yield_fixture_with_no_value ______________________________________________________________________
testdir = <Testdir local('/tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0')>
def test_yield_fixture_with_no_value(testdir):
testdir.makepyfile(
"""
import pytest
@pytest.fixture(name='custom')
def empty_yield():
if False:
yield
def test_fixt(custom):
pass
"""
)
expected = "E ValueError: custom did not yield a value"
result = testdir.runpytest()
result.assert_outcomes(error=1)
> result.stdout.fnmatch_lines([expected])
E Failed: nomatch: 'E ValueError: custom did not yield a value'
E and: '============================= test session starts =============================='
E and: 'platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7'
E and: 'rootdir: /tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0'
E and: 'collected 1 item'
E and: ''
E and: 'test_yield_fixture_with_no_value.py E [100%]'
E and: ''
E and: '==================================== ERRORS ===================================='
E and: '_________________________ ERROR at setup of test_fixt __________________________'
E and: ''
E and: '> ???'
E and: ''
E and: 'src/pluggy/callers/cythonized.pyx:14: '
E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E and: 'src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall'
E and: ' ???'
E and: 'src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall'
E and: ' ???'
E and: 'src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall'
E and: ' ???'
E and: 'src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall'
E and: ' ???'
E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E and: ''
E and: '> ???'
E and: 'E ValueError: custom did not yield a value'
E and: ''
E and: 'src/pluggy/callers/cythonized.pyx:43: ValueError'
E and: '=========================== short test summary info ============================'
E and: 'ERROR test_yield_fixture_with_no_value.py::test_fixt - ValueError: custom did...'
E and: '=============================== 1 error in 0.02s ==============================='
E remains unmatched: 'E ValueError: custom did not yield a value'
/home/ran/src/pytest/testing/python/fixtures.py:4346: Failed
---------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7
rootdir: /tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0
collected 1 item
test_yield_fixture_with_no_value.py E [100%]
==================================== ERRORS ====================================
_________________________ ERROR at setup of test_fixt __________________________
> ???
src/pluggy/callers/cythonized.pyx:14:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall
???
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E ValueError: custom did not yield a value
src/pluggy/callers/cythonized.pyx:43: ValueError
=========================== short test summary info ============================
ERROR test_yield_fixture_with_no_value.py::test_fixt - ValueError: custom did...
=============================== 1 error in 0.02s ===============================
I haven't looked at them, might be just something on pytest's side.
And here is a small pytest benchmark that I've used recently, which is not egrigously micro-benchmarky:
import pytest
@pytest.mark.parametrize("x", range(5000))
def test_foo(x): pass
it fires 100,408 hooks (I think). I run it under cProfile (best of 5, output is trimmed at 1s cumtime):
Results
Before:
10927116 function calls (10371875 primitive calls) in 9.935 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
365/1 0.007 0.000 9.947 9.947 {built-in method builtins.exec}
1 0.000 0.000 9.947 9.947 __main__.py:1(<module>)
1 0.000 0.000 9.841 9.841 __init__.py:146(console_main)
1 0.000 0.000 9.841 9.841 __init__.py:101(main)
100221/56 0.068 0.000 9.795 0.175 manager.py:90(_hookexec)
100221/56 0.152 0.000 9.795 0.175 manager.py:84(<lambda>)
100221/56 0.826 0.000 9.795 0.175 callers.py:157(_multicall)
100058/2 0.372 0.000 9.793 4.897 hooks.py:272(__call__)
1 0.000 0.000 9.553 9.553 main.py:249(pytest_cmdline_main)
1 0.000 0.000 9.553 9.553 main.py:188(wrap_session)
1 0.000 0.000 9.366 9.366 main.py:253(_main)
1 0.023 0.023 8.563 8.563 main.py:270(pytest_runtestloop)
5000 0.040 0.000 7.552 0.002 runner.py:97(pytest_runtest_protocol)
5000 0.039 0.000 6.668 0.001 runner.py:104(runtestprotocol)
15000 0.090 0.000 6.610 0.000 runner.py:203(call_and_report)
15002 0.110 0.000 4.980 0.000 runner.py:275(from_call)
15000 0.057 0.000 4.606 0.000 runner.py:224(call_runtest_hook)
15000 0.036 0.000 4.221 0.000 runner.py:239(<lambda>)
140057/120053 0.090 0.000 2.581 0.000 {built-in method builtins.next}
95050/95048 0.064 0.000 2.294 0.000 {method 'send' of 'generator' objects}
30000 0.077 0.000 1.269 0.000 capture.py:707(item_capture)
20003 0.029 0.000 1.122 0.000 contextlib.py:117(__exit__)
After:
10287357 function calls (9752107 primitive calls) in 9.441 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
366/1 0.006 0.000 9.453 9.453 {built-in method builtins.exec}
1 0.000 0.000 9.453 9.453 __main__.py:1(<module>)
1 0.000 0.000 9.345 9.345 __init__.py:146(console_main)
1 0.000 0.000 9.345 9.345 __init__.py:101(main)
100221/56 0.065 0.000 9.300 0.166 manager.py:90(_hookexec)
100221/56 0.079 0.000 9.300 0.166 manager.py:84(<lambda>)
100221/56 0.643 0.000 9.299 0.166 {built-in method pluggy.callers.cythonized._c_multicall}
100058/2 0.310 0.000 9.297 4.649 hooks.py:268(__call__)
1 0.000 0.000 9.056 9.056 main.py:249(pytest_cmdline_main)
1 0.000 0.000 9.056 9.056 main.py:188(wrap_session)
1 0.000 0.000 8.868 8.868 main.py:253(_main)
1 0.023 0.023 8.073 8.073 main.py:270(pytest_runtestloop)
5000 0.040 0.000 7.108 0.001 runner.py:97(pytest_runtest_protocol)
5000 0.040 0.000 6.250 0.001 runner.py:104(runtestprotocol)
15000 0.091 0.000 6.190 0.000 runner.py:203(call_and_report)
15002 0.109 0.000 4.815 0.000 runner.py:275(from_call)
15000 0.057 0.000 4.446 0.000 runner.py:224(call_runtest_hook)
15000 0.035 0.000 4.056 0.000 runner.py:239(<lambda>)
45007 0.033 0.000 1.901 0.000 {built-in method builtins.next}
30000 0.076 0.000 1.268 0.000 capture.py:707(item_capture)
20003 0.035 0.000 1.133 0.000 contextlib.py:117(__exit__)
So it's a 5% improvement. Note that cProfile slows down the total execution time but relatively it should be about right.
IMO, 5% is probably not enough to be worth it, given that Cython, C extensions, wheels etc. are a huge headache all around. But if the Cython implementation can be sped up further I'll be happy to check again.
So it's a 5% improvement.
Yeah nothing to write home about.
IMO, 5% is probably not enough to be worth it, given that Cython, C extensions, wheels etc. are a huge headache all around.
Agreed, at least not as the default packaging. I think that if we want to include it as an optional dependency once we can get a better speedup that's something to consider.
What's the raw execution time on cryptography for example, or a test suite with 40k tests
Check this out.
@oremanj just pointed to mypyc in the trio gitter which might be worth investigating as well for other modules in pytest and pluggy.
I know @bluetech is gonna luv the type hints :)