Support Arbitrarily Repository Metrics

[fkorotkov]

Description

In addition to #93 it will be nice to expose an ability for users to expose arbitrary metrics like code coverage, benchmark score, amount of TODOs, etc.

Context

Cirrus can comment on PRs with comparison of such stats with the base of the PR. Plus expose badges.

Each stat should have properties like:

• unit to define if it's a percent or an absolute number
• merge_strategy to define how to merge stats with the same name from different tasks. Either override or average (which might be useful for benchmarking: run couple of tasks in parallel and take the average).

Syntax might look like this:

on_success:
  metric:
    name: performance
    unit: seconds
    merge_strategy: average
    value_script: cat benchmarking/performance.txt

[RDIL]

Yes, this sounds so cool!

[tobim]

on_success would be a key under task, right? If so, the metric name should be concatenated with the task name by default, and merging of metrics across tasks should only be done if requested.

[tobim]

I recently found https://seriesci.com/ for an app that implements something like this in an orthogonal way.

[fkorotkov]

Interesting! Will keep an eye on @seriesci! Thanks for sharing!

I was thinking about scoping metrics by build (aka SHA). That's why there is merge_strategy to merge metrics of the same name produced by different tasks within a build (you can run benchmark several times and take an average as a result).