project-infra icon indicating copy to clipboard operation
project-infra copied to clipboard

flake-stats: add number of merged PRs per day to box

Open dhiller opened this issue 5 months ago • 6 comments

Is your feature request related to a problem? Please describe: The flake-stats report1 is our main tool to look at flaky tests during the SIG-CI meetings currently. It's been proven extremely useful to get a quick overview about where things go wrong. This is achieved by the overviews that aggregate failures per day and per lane.

Now, sometimes it seems that there are a lot of failures that have occurred on one day, solely being caused by the fact that a lot of PRs have been merged on that day - reason of this is that the sum of failures per day is taken from the flakefinder report which aggregates all the failures of a merged PR towards the day it was merged, and not towards the day the failure itself occurred.

[!NOTE] Example: a PR that was created on Dec 21st and is merged on Jan 1st, during that period it's test lanes had been run several times, and there were some unit test failures on those runs. Now, all the failures are accounted towards the day the PR is getting merged, so that exactly that day looks as if a lot of failures occurred at that day - which again is not the case.

Describe the solution you'd like: By adding the number of merged PRs to a day's numbers we can better spot anomalies - i.e. a high number of test failures even though the number of merged PRs is low might indicate that a flaky or unstable test got introduced.

Thus we want to show the number of PRs that have been merged on the day - added to the box that shows the failures for that day. Sketch:

Mon, 26 Aug 2024
Failures 42
PRs merged 10

Additional context:

  • flake-stats code: https://github.com/kubevirt/project-infra/tree/main/robots/cmd/flake-stats
  • flake-stats Go Template location: https://github.com/kubevirt/project-infra/blob/a609d2f66961a57e6a447e29151f44bf6cf14464/robots/cmd/flake-stats/flake-stats.gohtml#L32
  • The number of PRs (also the actual PR numbers) is available from the json element "prNumbers"

dhiller avatar Sep 09 '24 12:09 dhiller