coveragepy
coveragepy copied to clipboard
Record custom metadata for contexts
Is your feature request related to a problem? Please describe. We currently run coverage in CI jobs, with some jobs such as unit tests being run in the pipeline triggered by the push, but other jobs that are more expensive to run are run less frequently, in separate pipelines. We are using coverage to find code that is not being used, and track which jobs / tests are using which parts of the code. One difficulty we've run into is tracking which version of the code (git commit hash) each job (and therefore each coverage file) uses. It would also sometimes be useful to be able to trace a context back to the CI job that generated it.
Describe the solution you'd like Allow extra metadata fields to be recorded when recording coverage. In our case, it would be a few fields such as the git commit hash, and job URL. This could be passed via command line, using something like
coverage run --context=foo --context-meta=git-hash=01234abc --context-meta=job-url=http://example.com/...
or in code using something like
switch_context('foo', git_hash='01234abc', job_url='http://example.com/...')
or
switch_context('foo', {'git-hash': '01234abc', 'job-url': 'http://example.com/...'})
It should allow any strings as the key and value.
Extra function/s would also be required in CoverageData class, to read the metadata. It could also be useful to add metadata later, such as adding details of a failed test / thrown exception, or a commit that was later deployed to production.
Describe alternatives you've considered Storing the metadata separately, such as in an json file. However, that will make combining context files tricky, having to handle the json files too.
Additional context Other metadata could automatically be recorded, such as start / end timestamp