WIP, ENH: add docstring cov to html

[tylerjereddy]

One of the items on the wishlist for testing docstrings in NumPy / SciPy is the ability to report "coverage" for those docstrings that are probed by doctesting infrastructure. In particular, I had in mind the idea that labels in the html coverage reports that one often finds as part of the CI for large OSS scientific projects might be expanded to include hits on docstrings.

This lead to me to a simple example on StackOverflow by @rhettinger, which really got me excited even more. However, I noticed that executing the exact same code and trying to display with coverage html effectively ignores the docstring using both stable and master branch of coveragepy. I dug a little deeper and noticed that coverage annotate actually has a much shorter / simpler loop for doing the labelling vs. html, and I figured I'd take a look at the html assignment of lines to label classes, and see if I could get it to do what I want.

This PR has a draft of what I had in mind that I initially thought was working, but actually seems to be not quite right yet. There are probably various ecosystem considerations & things like trying to fuse these results with results that don't have docstrings labelled, etc. But if I I can't get this "simple" case to work, the prospects for expanding the "feature" usage to large packages wouldn't be great.

With this feature branch and my .coveragerc initially looking like this:

[html]
docstring_display = True

with the original example code from StackOverflow we get the docstring now highlighted along with the source itself that was ultimately executed by doctest:

and with a doctest SKIP directive added in to Raymond's example, another design question comes up--would it be possible / worth the trouble to have a separate highlight color / category for docstring examples that are not executed? (notice that the source proper is still correctly marked as not executed, since the docstring didn't get executed, although the docstring was "considered but skipped" so perhaps technically "covered"):

So, as I was writing this up, I thought "hey this is working just like the annotate case so must be good," but then I commented out the doctest execution and it seems the docstring lines are just getting labelled as a fallback, not because the tracing infrastructure knows that they are getting executed:

Ok, so maybe I got excited a little too early. That got me curious about Raymond's original example with annotate. If I turn off the docstring execution there by commenting out the doctest.testmod() line I get a result suggesting that docstring hits are not actually meaningful results from coveragepy tracing / interpretation infrastructure:

> def linear(a, b):
>     ''' Solve ax + b = 0
  
>         >>> linear(3, 5)
>         -1.6666666666666667
  
>     '''
!     if a == 0 and b != 0:
!         raise ValueError('No solutions')
!     return -b  / a

This suggests to me that there may be another "meta" level to the tracing if we want to handle doctest probing of those string lines as "hits." That might be split into two categories:

1. was the string "probed" by doctest infrastructure?
2. was a line in that string exec(compile()) run by doctest or subclassed doctest infrastructure in a library like NumPy / SciPy?

[nedbat]

Sorry I've left this for so long: as I understand it, the goal is to mark docstrings as executed if doctest executes them? I'm not sure what hooks doctest gives us for knowing which are executed. Or do the line numbers get reported as executed when doctest runs them? I'll have to do some experiments.

In the examples you show, I'm not sure why line 2 is marked as executed? It's not executable code, which makes me think the entire docstring is marked executed even if only part of it is.

[tylerjereddy]

Sorry I've left this for so long: as I understand it, the goal is to mark docstrings as executed if doctest executes them?

Yes, that would be cool, but as you note it is not immediately clear how to trace through the "meta-execution" that doctest in CPython does. I think I saw an explicit exec when I took a peek inside doctest, but I don't know if that's enough to trace through.

[ProsperousHeart]

Is this still a valid PR? I know it's in draft format, but it's a few years old at this point with several updates since then

[tylerjereddy]

I think the idea is good and still not supported, but this PR is probably useless beyond requesting the feature at this point.