sig-release icon indicating copy to clipboard operation
sig-release copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

Define and collect input metrics about Kubernetes releases

Open saschagrunert opened this issue 3 years ago • 23 comments

Target is to define a set of metrics around Kubernetes releases to elaborate on the reduced release cadence. This discussion came up during the KEP implementation phase. The planned survey outcome can help us to interpret the raw data later on, too.

The following questions have to be resolved before starting to collect the metrics:

  • Which metrics do we want to collect? For example, the number of …
    • exceptions requests
    • PR merged after code freeze
    • tests fixed during test freeze
    • issues open during code/test freeze
    • backport per minor release
    • requests for exceptions after code freeze
  • How do we want to collect the data? It could be possible that the release team leads collect the metrics before every retrospective and present them there. This would allow us to gather feedback within the retrospective before interpreting the data or correlating it to anything.
  • How to track the metrics over multiple releases? Making assertions based on the data needs multiple releases as input. We should take care as a SIG to keep track of them and evaluate periodically.

/priority important-longterm /cc @aojea /help

saschagrunert avatar Apr 19 '21 07:04 saschagrunert

We have to start defining which areas we want to measure, then define the metrics and the actions associated to each metrics, (I'm making up the examples), i.e.:

Quality:

  • CI reliability: Description: This measure the CI health of the project based on some specific jobs considered blocking, this is a daily metric Metrics: The metrics is obtained from https://testgrid.k8s.io/sig-release-master-blocking , based on number of failed jobs Threshold: 99% Actions: Immediate action from all SIGs to fix the failing jobs
  • Stability: Description: Measure the stability of the project based on the number of backports to the stable branches: Metrics: Cherry-picks approved per minor release Threshold: this should be trend based, going higher means we are doing more backports, and it may indicate we are neglecting quality Actions: ....

Release:

  • Exception requests: Description: It measures the project development speed vs current defined timelines. This is a per release metrics. Metrics: The metrics is obtained based on the exceptions request to include features after code free time. The information can be clustered if the process use some kind of form, i.e. what is the reason to request an exception: feature to big, bad planning, .... Threshold: 10% , if the release expected to have 50 features, only 5 exceptions max are expected Actions: If there are many exception request it is a clear symptom that the developer speed/planning is not aligned to the release timeline, it should require wide discussion within the project (maybe scalated to steering ?? )....

ideally, everything should be automated and you just can build a dashboard with that, per example, the exception requests instead of an email can be an issue or a PR, or a form, so you can automate it.

aojea avatar Apr 19 '21 07:04 aojea

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot avatar Jul 18 '21 08:07 fejta-bot

@aojea are you still planning to work on this? :)

saschagrunert avatar Jul 19 '21 07:07 saschagrunert

@aojea are you still planning to work on this? :)

Sorry, I was just sharing some experience and things that worked for me in response to the KEP about the release cadence. I really don't have time ...

aojea avatar Jul 19 '21 07:07 aojea

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Aug 18 '21 07:08 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-triage-robot avatar Sep 17 '21 08:09 k8s-triage-robot

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Sep 17 '21 08:09 k8s-ci-robot

Would it make sense to title this "Define and collect input metrics about Kubernetes releases"? Currently it feels scoped in that way. It would be very useful to additionally have output usage metrics (eg: download counts of different artifacts during specific time spans).

tpepper avatar Oct 07 '21 21:10 tpepper

It would be very useful to additionally have output usage metrics (eg: download counts of different artifacts during specific time spans).

we should be careful with those metrics to avoid falling into the marketing/populism, downloads of artifacts is commonly abused by CI systems ... if you just can differentiate users from bots :thinking: , This problem is similar as counting number of contributors and including people that fixed a typo in one year :/

aojea avatar Oct 07 '21 22:10 aojea

https://github.com/kubernetes/sig-release/issues/1372#issuecomment-767108296 represents a way to get at less game-able data perhaps.

Simple stats gaming bots are possible, but I wouldn't consider CI as inherently a problem. I'm actually interested in seeing the bulk CI usage too. If people are still doing a lot of CI on 1.9 that might be informative.

And I'm really interested in the zeros. Which of our RPM and deb variants have zero downloads?

tpepper avatar Oct 07 '21 22:10 tpepper

agree

aojea avatar Oct 07 '21 22:10 aojea

Sounds good, created https://github.com/kubernetes/sig-release/issues/1723 for elaboration.

saschagrunert avatar Oct 08 '21 07:10 saschagrunert

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 06 '22 08:01 k8s-triage-robot

/remove-lifecycle stale

saschagrunert avatar Jan 10 '22 09:01 saschagrunert

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Apr 10 '22 10:04 k8s-triage-robot

/remove-lifecycle stale

saschagrunert avatar Apr 11 '22 07:04 saschagrunert

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jul 10 '22 07:07 k8s-triage-robot

/remove-lifecycle stale

palnabarun avatar Jul 14 '22 12:07 palnabarun

I'm not sure what statistics you get from your artifact registry, but here are a few thoughts on the usage side

  • gcr.io VS k8s.io registry during the transition
  • count by client used / count of unique downloads by IP or /24
  • it would be really interesting to know how many are upgrading for each patch / minor vs skipping (this could feed into release cycle discussions)

For within-release management:

  • number of KEPs for release at each cycle checkpoint
    • where do KEPs drop out of the cycle? Should this move left or right?
  • number of days with "green" CI
  • number of bug fixes / regressions detected VS fixed during the cycle
  • (post release) number of bug fixes and which patch they were fixed in

evankanderson avatar Jul 18 '22 15:07 evankanderson

  1. Number of Enhancements dropped from the release, with a reason category for why they were dropped. Defer to enhancements team to come up with 2-6 categories for dropping enhancements (e.g. feature cancelled permanently, author incommunicado, etc.)
  2. Number/percent of release team shadows who completed the release work (i.e. didn't drop out).
  3. Number of issues reported based on alpha and beta releases, by day (i.e. is anyone actually testing these?)

jberkus avatar Jul 25 '22 22:07 jberkus

+1 to calls for metrics on # of regressions, kep status’ (current and future releases; eg how many KEPs not GA yet, how many went to GA, etc), and team shadow metrics

parispittman avatar Jul 27 '22 17:07 parispittman

The new project board for enhancements would help us to gather the metrics in a more automated way. I think we should give the board more time to evolve while keeping the necessity for those metrics in mind.

cc @leonardpahlke

Referring discussion: https://kubernetes.slack.com/archives/C2C40FMNF/p1662537798532949?thread_ts=1662477434.202879&cid=C2C40FMNF

saschagrunert avatar Sep 07 '22 08:09 saschagrunert

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Feb 08 '23 02:02 k8s-triage-robot