Fine grained differentiation of coverage context

[tysonclugg]

Is your feature request related to a problem? Please describe. It's easy to increase coverage using some testing strategies (eg: SnapshotTest), but coverage stats are skewed for other strategies (eg: BDD) where coverage is often far lower. I'd like to have coverage not just report the high level coverage which could well be 100% across all test strategies, but to report finer grained coverage depending on execution context.

The problem I'm having is that I can't filter out subsets of execution context from the test results.

Describe the solution you'd like To be able to provide additional context from within the code under test so that coverage can be differentiated in a more meaningful way. Perhaps something like this:

import coverage
import django.test
import snapshottest


class APITestCase(snapshottest.TestCase, django.test.TestCase):
    def test_api_me(self):
        """Testing the API for /me"""
        with coverage.extra_context("SnapshotTest"):
            my_api_response = self.client.get('/me')
            self.assertMatchSnapshot(my_api_response)

Describe alternatives you've considered

• Running different subsets of the entire test suite with different contexts results in expensive setup/teardown tasks being run multiple times.
• Perhaps hooking into unittest.TestCase.subTest could yield results without depending on Coverge.py specific APIs.
• There is also the opportunity to integrate with the OpenTelemetry Python API to provide a much richer set of execution context, once again without Coverage.py specific APIs.

Additional context I'd like to ensure certain modules (eg: **/viewsets.py) have very high coverage within particular execution contexts.

[nedbat]

Have you looked at the Measurement Context section of the docs? I don't think it has what you want yet, but it's the closest thing at the moment.

[tysonclugg]

I'm very much aware of the measurement context features. The issue is that I want more context that what is currently possible. A fundamental limitation here is that each arc references a single context.

My guess at a possible solution would be to extend the FileTracer class to allow capture of additional metadata which is then recorded against each arc, perhaps in a new JSON/text column. This could result in an opentelemetry-python / coverage plugin that roughly does something similar to this:

import coverage
from opentelemetry import trace

class OpenTelemetryFileTracer(coverage.FileTracer):
    def get_arc_metadata(self, frame):
        """Generate metadata to be recorded with each coverage arc."""
        span_context = trace.find_span_for_frame(frame).get_span_context()
        return {
            "trace_id": trace.format_trace_id(span_context.trace_id),
            "span_id": trace.format_span_id(span_context.span_id),
        }