coveragepy
coveragepy copied to clipboard
nedbat
Coverage measurement fails on code containing os.exec* methods
Originally reported by Anonymous
I recently tried to measure coverage of a program that calls os.execvpe, essentially causing the process to be replaced by a different one. This did not record any coverage information at all.
The reason of course is that os.execvpe does not return, so there is no opportunity to call coverage.stop() and coverage.save() as is done if e.g. an exception is thrown. I'd suggest this method could be "monkey-patched" so that such code can be inserted before it. (and also the other 7 os.exec* methods of course)
- Bitbucket: https://bitbucket.org/ned/coveragepy/issue/43
- This issue had attachments: execfunctions.bundle. See the original issue for details.
Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)
Implementing a solution like this now appears to be difficult, as the "coverage" object is not necessarily stored in a global variable any more.
I resorted to the following in my code, which does not require patching coverage at all, but only works under Python 2 and relies on an undocumented feature of Python's atexit module:
#!python
import atexit
for func, args, kw in atexit._exithandlers:
if func.__module__.startswith("coverage."):
func(*args, **kw)
Clearly not ideal but about the only thing I could think of to get my coverage measurement working with the latest version.
Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)
Looking at this again. There are other circumstances where no coverage information is produced because the atexit handlers are not called:
- When processes are terminated by a signal
- When control is not returned to the Python interpreter. I've run into this in several circumstances under Jython, where Java code shuts down the JVM without returning control to Jython or calling any exit handlers.
In the light of this, I'm wondering if a generic, if slightly clumsy, solution would be simply to provide a "process_shutdown" method (in a similar way to process_startup) so that this external handler could be called explicitly in any of these circumstances.
Then it's easy to add code like
#!python
try:
import coverage
coverage.process_shutdown()
except:
pass
and call it in signal handlers, just before os.exec* or in a JVM exit handler to work around all these cases.
Would be better of course if it was detected automatically, but this will avoid the evils of monkey patching and is more generic than the code I posted here. Can provide a patch if you agree to the solution.
For anyone looking for Geoff's changes:
diff -r [f7d26908601c (bb)](https://bitbucket.org/ned/coveragepy/commits/f7d26908601c) -r [f3a76cf7aa00 (bb)](https://bitbucket.org/ned/coveragepy/commits/f3a76cf7aa00) coverage/control.py
--- a/coverage/control.py Sun Nov 07 19:45:54 2010 -0500
+++ b/coverage/control.py Mon Nov 15 21:36:31 2010 +0100
@@ -360,6 +360,14 @@
self._harvested = False
self.collector.start()
+ os.execvpe = self.intercept(os.execvpe)
+
+ def intercept(self, method):
+ def new_method(*args, **kw):
+ self.stop()
+ self.save()
+ method(*args, **kw)
+ return new_method
def stop(self):
"""Stop measuring code coverage."""
and then:
diff -r [f3a76cf7aa00 (bb)](https://bitbucket.org/ned/coveragepy/commits/f3a76cf7aa00) -r [ba05ad03668e (bb)](https://bitbucket.org/ned/coveragepy/commits/ba05ad03668e) coverage/control.py
--- a/coverage/control.py Mon Nov 15 21:36:31 2010 +0100
+++ b/coverage/control.py Mon Nov 15 22:37:22 2010 +0100
@@ -359,15 +359,13 @@
self.omit_match = FnmatchMatcher(self.omit)
self._harvested = False
+ for funcName in [ 'execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe' ]:
+ newFunc = self.intercept(getattr(os, funcName))
+ setattr(os, funcName, newFunc)
self.collector.start()
- os.execvpe = self.intercept(os.execvpe)
- def intercept(self, method):
- def new_method(*args, **kw):
- self.stop()
- self.save()
- method(*args, **kw)
- return new_method
+ def intercept(self, method):
+ return StopCoverageDecorator(self, method)
def stop(self):
"""Stop measuring code coverage."""
@@ -612,6 +610,21 @@
return info
+class StopCoverageDecorator:
+ inDecorator = False
+ def __init__(self, cov, method):
+ self.cov = cov
+ self.method = method
+
+ def __call__(self, *args, **kw):
+ if not StopCoverageDecorator.inDecorator:
+ StopCoverageDecorator.inDecorator = True
+ self.cov.stop()
+ self.cov.save()
+ self.method(*args, **kw)
+ StopCoverageDecorator.inDecorator = False
+
+
def process_startup():
"""Call this at Python startup to perhaps measure coverage.
I haven't applied the patch yet, because I'd only heard one request for it (this ticket), and I am averse to monkeypatching. But I now have a second request, and this is a fairly small patch.
Interestingly, if execvpe would execute the atexit-registered handlers before changing the process over, it would just work. I created http://bugs.python.org/issue16822 to request Python to be fixed.
Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)
Patch attached against current trunk. Could be simplified if it assumed that these functions called each other internally, which they do currently.
Tested with a small program as follows
#!python
import os
print "First program..."
os.execvp("./prog2.py", [ "./prog2.py", "-x" ])
which I can now get coverage information out of.
Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)
I have a patch for this one also if you're interested.
Original comment by Geoff Bache (Bitbucket: geoffbache, GitHub: Unknown)
Sorry, didn't mean to report this anonymously.
CC. Still affected in ``` Python 3.7.3 (default, Jul 25 2020, 13:03:44)
import coverage coverage.version '4.5.2'
Here's another hack that seems to work for Python 3, at least with coverage run: it walks the stack to find the Coverage object ...
#!python
if 'COVERAGE_RUN' in os.environ and sys.gettrace() is not None:
def _find_cov():
from traceback import walk_stack
for (f, _) in walk_stack(None):
for obj in f.f_locals.values():
if (obj.__class__.__name__ == 'CoverageScript' and
obj.__module__ == 'coverage.cmdline'):
return obj.coverage
cov = _find_cov()
if cov is not None:
cov.stop()
cov.save()
Nothing new has happened. Can you tell us more about how and why os.exec is involved? It's a difficult thing to put a leash around with certainty.
I have a function that calls os.execv, which I'm testing with pytest, let's call it func_execv.
My testing code looks as follows (to prevent pytest crashing when the process is replaced). Minimal reproduction:
from multiprocessing import Process
import os
def func_execv(cmd, args):
os.execvp(cmd, args)
def test_func_execv(capfd):
cmd = "echo"
args = [cmd, "test"]
p = Process(target=func_execv, args=(cmd, args))
p.start()
p.join()
assert "test" in capfd.readouterr().out
Run this file with pytest file.py --cov in an environment with pytest and pytest-cov.
I have to execute the func_execv inside Process because otherwise pytest crashes due to process exit.
The code in func_execv is not covered. coverage report -m:
Name Stmts Miss Cover Missing
---------------------------------------------
scratch_51.py 11 1 91% 6
---------------------------------------------
TOTAL 11 1 91%
Do the steps for subprocess measurement help? https://coverage.readthedocs.io/en/7.3.2/subprocess.html
I've added the suggested line to _virtualenv.pth and also set the env variable COVERAGE_PROCESS_START, but it didn't help.
Can you suggest a basic setup with the minimum reproduction example where it would work?