coveragepy
coveragepy copied to clipboard
nedbat
Support coverage plugins with sys.monitoring
Is your feature request related to a problem? Please describe.
Running coverage.py on Cython code with sys.monitoring in Py3.13 disables the Cython coverage plugin:
https://github.com/nedbat/coveragepy/blob/7f336228d78d8338239d4da4d4de3ade0abd0901/coverage/control.py#L565-L568
Describe the solution you'd like Plugins should work the same with the old tracing and new monitoring infrastructure.
Additional context I'm currently working on getting Cython to use the monitoring C-API, but found that coverage.py doesn't support both together as it stands.
Hi, sorry to let this sit. What plugin behavior in particular are you looking for? The key idea of sys.monitoring that makes it fast is to disable an event once it's fired, but some plugin methods would need the events to be re-enabled. This is extra choreography that I haven't really thought through yet, and I hope it doesn't completely nullify the speed benefits.
Our coverage plugin is here: https://github.com/cython/cython/blob/master/Cython/Coverage.py
We don't need access to events, we just do file lookups and line matching.
With Python 3.14 this now just crashes:
$ python --version
Python 3.14.0rc2
$ coverage --version
Coverage.py, version 7.10.6 with C extension
Full documentation is at https://coverage.readthedocs.io/en/7.10.6
$ cython --version
Cython version 3.1.3
$ PYTHONPATH=src coverage run --rcfile=.coveragerc.setuptools -m flint.test
~/python-flint/.venv_314/lib/python3.14/site-packages/coverage/control.py:619: CoverageWarning: Plugin file tracers (Cython.Coverage.Plugin) aren't supported with SysMonitor
self._warn(
Traceback (most recent call last):
File "/stuff/current/active/python-flint/.venv_314/bin/coverage", line 10, in <module>
sys.exit(main())
~~~~^^
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/cmdline.py", line 1135, in main
status = CoverageScript().command_line(argv)
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/cmdline.py", line 834, in command_line
return self.do_run(options, args)
~~~~~~~~~~~^^^^^^^^^^^^^^^
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/cmdline.py", line 1023, in do_run
runner.run()
~~~~~~~~~~^^
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/execfile.py", line 174, in run
self._prepare2()
~~~~~~~~~~~~~~^^
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/execfile.py", line 139, in _prepare2
pathname, self.package, self.spec = find_module(self.modulename)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/execfile.py", line 49, in find_module
spec = importlib.util.find_spec(modulename)
File "<frozen importlib.util>", line 90, in find_spec
File "/stuff/current/active/python-flint/src/flint/__init__.py", line 1, in <module>
from .pyflint import ctx
File "src/flint/pyflint.pyx", line 1, in init flint.pyflint
"""
File "src/flint/flint_base/flint_base.pyx", line 1, in init flint.flint_base.flint_base
from flint.flintlib.types.flint cimport (
File "src/flint/flint_base/flint_context.pyx", line 203, in init flint.flint_base.flint_context
cdef FlintContext thectx = FlintContext()
File "src/flint/flint_base/flint_context.pyx", line 12, in flint.flint_base.flint_context.FlintContext.__init__
def __init__(self):
File "/stuff/current/active/python-flint/.venv_314/lib/python3.14/site-packages/coverage/sysmon.py", line 364, in sysmon_py_start
sys_monitoring.set_local_events(self.myid, code, local_events)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: cannot instrument shim code object '__init__'
Is that a different issue?
That looks like a malformed code object. What is code on the line sys_monitoring.set_local_events(self.myid, code, local_events)?
My guess is that Cython code is firing a PY_START event, then coverage is calling sys_monitoring.set_local_events(self.myid, code, local_events) on the fake code object created by Cython.
We discussed how Cython could provide a "code-like" object, so that sys_monitoring.set_local_events could handle cases like this.
https://github.com/python/cpython/issues/117087.
Until that is resolved, I think coverage and cython will need to find a way to communicate. We could add some sort of hook for code generators like Cython to intercept the setting of local events, but that obviously won't happen before 3.15.
SystemError: cannot instrument shim code object '__init__'
This happens because Cython's code objects have empty byte code: https://github.com/python/cpython/blob/f19f1d8563fb3abbb673812f16e2be5f10af42e4/Python/instrumentation.c#L2043-L2046
I considered filling the byte code with NOPs or similar to give them some kind of meaning, but the byte code is version specific and I found no easy way to do this across CPython versions.
According to this code:
https://github.com/python/cpython/blob/f19f1d8563fb3abbb673812f16e2be5f10af42e4/Python/instrumentation.c#L2043-L2046
it could help to fill (or start) it with RESUME opcodes, although I also doubt that that'd be portable.
Cython could reset codeobj->_co_firsttraceable to 0 after the creation, though. Here is a patch against Cython (master/3.2) that should do that:
diff --git a/Cython/Utility/ModuleSetupCode.c b/Cython/Utility/ModuleSetupCode.c
index c342c9c66..48c78a8e1 100644
--- a/Cython/Utility/ModuleSetupCode.c
+++ b/Cython/Utility/ModuleSetupCode.c
@@ -2597,6 +2597,10 @@ static PyObject* __Pyx_PyCode_New(
(__PYX_LIMITED_VERSION_HEX >= (0x030b0000) && line_table) ? line_table : EMPTY(bytes)
);
+ #if CYTHON_COMPILING_IN_CPYTHON && PY_VERSION_HEX >= 0x030c00A1
+ ((PyCodeObject*) code_obj)->_co_firsttraceable = 0;
+ #endif
+
done:
Py_XDECREF(code_bytes);
#if CYTHON_AVOID_BORROWED_REFS
@oscarbenjamin could you try this on your side?
that obviously won't happen before 3.15.
Anything we do about this in CPython would only fix the future and would require work-arounds for the existing CPython versions.
@oscarbenjamin could you try this on your side?
Yes, that does handle the immediate error.
With Cython master (93afd3c980907c9350c2bc1bcd9aa639a31a2ee9) I see the SystemError: cannot instrument shim code object '__init__' error.
With _co_firsttraceable = 0 that error goes away and the tests can run under coverage (although one fails because of some other change in cython master).
The coverage measurement still does not work though. This is with Python 3.12 and Cython 3.1.3:
$ coverage report --rcfile=.coveragerc.setuptools
Name Stmts Miss Cover
--------------------------------------------------------------
src/flint/__init__.py 38 0 100%
src/flint/flint_base/__init__.py 0 0 100%
src/flint/flint_base/flint_base.pyx 495 70 86%
src/flint/flint_base/flint_context.pxd 8 4 50%
src/flint/flint_base/flint_context.pyx 53 11 79%
src/flint/flintlib/__init__.py 0 0 100%
src/flint/flintlib/functions/__init__.py 0 0 100%
src/flint/flintlib/types/__init__.py 0 0 100%
src/flint/functions/__init__.py 0 0 100%
src/flint/functions/showgood.pyx 61 8 87%
...
I cut the output short but you can see that .pyx and .pxd files are listed.= in the coverage statistics.
This the output with Python 3.14.0rc2 and Cython master and _co_firsttraceable = 0:
$ coverage report --rcfile=.coveragerc.setuptools
Name Stmts Miss Cover
--------------------------------------------------------------
src/flint/__init__.py 38 0 100%
src/flint/flint_base/__init__.py 0 0 100%
src/flint/flintlib/__init__.py 0 0 100%
src/flint/flintlib/functions/__init__.py 0 0 100%
src/flint/flintlib/types/__init__.py 0 0 100%
src/flint/functions/__init__.py 0 0 100%
src/flint/test/__init__.py 0 0 100%
src/flint/test/__main__.py 85 17 80%
src/flint/test/test_all.py 3763 14 99%
src/flint/test/test_docstrings.py 54 14 74%
src/flint/types/__init__.py 0 0 100%
src/flint/typing.py 17 0 100%
src/flint/utils/__init__.py 0 0 100%
src/flint/utils/flint_exceptions.py 10 0 100%
--------------------------------------------------------------
TOTAL 3967 45 99%
That is only the .py files so none of the Cython code has been measured.
I'm a bit surprised that you are getting meaningful data for Cython, as coverage's call to sys.monitoring.set_local_events will have no effect.
I can fix sys.monitoring.set_local_events by getting PyMonitoring_EnterScope to account for local events.
I doubt it will get into 3.14 though, as we are into RC phase.
I think we can get a fix to not raise a SystemError if the code size is 0 into 3.14.
What is the Py_SIZE() of Cython code objects? If you set _co_firsttraceable = 0 then instrumentation could trash whatever Cython stores in the bytecode array and could crash (or worse) if those bytecodes are anything other legal code.
I'd recommend setting Py_SIZE() = 1, and making the first "instruction" in the code array a NOP, as well as setting _co_firsttraceable = 0.
For 3.15 we can allow Py_SIZE() == 0 and treat it as a marker for artificial code objects, so that we can handle monitoring for them properly.
What is the
Py_SIZE()of Cython code objects?
The "byte code" in Cython's code objects is currently a zero-initialised byte string that is a little longer than the PEP-626 line table to make CPython accept the line table. https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L2562-L2578
I initially wanted to fill it with NOP-codes but found that to be difficult due to version/platform specific differences in the byte code (and more generally my persistent failure to bring the byte sequence into a format that CPython would accept). Thus, I left it as an all-NUL string.
For 3.15 we can allow
Py_SIZE() == 0and treat it as a marker for artificial code objects, so that we can handle monitoring for them properly.
That seems quite reasonable, and would allow disabling the "line table must match byte code" check.
Looks like https://github.com/nedbat/coveragepy/blob/8827634/coverage/env.py#L49 is causing Cython's CI failures on 3.14: https://github.com/cython/cython/pull/7197.
