coveragepy icon indicating copy to clipboard operation
coveragepy copied to clipboard

Record custom metadata for contexts

Open Zeckie opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe. We currently run coverage in CI jobs, with some jobs such as unit tests being run in the pipeline triggered by the push, but other jobs that are more expensive to run are run less frequently, in separate pipelines. We are using coverage to find code that is not being used, and track which jobs / tests are using which parts of the code. One difficulty we've run into is tracking which version of the code (git commit hash) each job (and therefore each coverage file) uses. It would also sometimes be useful to be able to trace a context back to the CI job that generated it.

Describe the solution you'd like Allow extra metadata fields to be recorded when recording coverage. In our case, it would be a few fields such as the git commit hash, and job URL. This could be passed via command line, using something like

coverage run --context=foo --context-meta=git-hash=01234abc --context-meta=job-url=http://example.com/...

or in code using something like

switch_context('foo', git_hash='01234abc', job_url='http://example.com/...')

or

switch_context('foo', {'git-hash': '01234abc', 'job-url': 'http://example.com/...'})

It should allow any strings as the key and value.

Extra function/s would also be required in CoverageData class, to read the metadata. It could also be useful to add metadata later, such as adding details of a failed test / thrown exception, or a commit that was later deployed to production.

Describe alternatives you've considered Storing the metadata separately, such as in an json file. However, that will make combining context files tricky, having to handle the json files too.

Additional context Other metadata could automatically be recorded, such as start / end timestamp

Zeckie avatar Jan 26 '23 23:01 Zeckie