kubernetes
testing strategy: add policy for presubmits
This was triggered by https://github.com/kubernetes/test-infra/pull/33463#issuecomment-2348289131 and is part of an effort to document best practices. The guidelines are cut-and-pasted from a discussion on Slack were we agreed that that they seem reasonable and should be documented publicly.
The part about blocking presubmits is based on https://kubernetes.slack.com/archives/C2C40FMNF/p1734418617113169?thread_ts=1734417601.687079&cid=C2C40FMNF
The guidelines are intentionally in a sub-section because then we can provide a deep link whenever this comes up in a test-infra PR review.
/assign @aojea @michelle192837
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: pohly
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~contributors/devel/sig-testing/OWNERS~~ [pohly]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
We also need guidance about that fact that you shouldn't rely on path triggering to accurately reflect the inputs to something under test.
IE you have a go tool and you have some extra tests for it that trigger on cmd/tool/.* except that misses the go mod, go sum, build scripts, go version, and so on. Getting that right is extremely error prone (you are re-inventing bazel target graphs but with handwritten regex) and a reason to prefer manual triggering instead.
I don't generally recommend this sort of non-blocking path based job and I'm struggling to think of a successful counter example
Is this really a best practice?
/hold
I'm out for the holidays but will try to draft a comment on that test-infra PR soon.
this approach is extremely error prone and has not gone well in the past for any attempt I've seen
@BenTheElder I've tried to address both points in https://github.com/kubernetes/community/pull/8196#issuecomment-2503957800 with https://github.com/kubernetes/community/compare/a072efd375511540ba0e8aba4845f4d7d133f0ad..e113ccd92f78a8c26a3166799c6aa7fc9352565d
this approach is extremely error prone and has not gone well in the past for any attempt I've seen
I'm quite happy with this approach for DRA. It has served us (= the DRA developers) pretty well. I'm aware that this came at a cost for other developers (while transitioning between API versions, we had to break the jobs for master to make them work in the PR under review, and that showed up as test failures in other PRs), but I hope the feature justified that.
If that hadn't been resolved quickly, the jobs should have been disabled because they didn't meet the criteria outlined in this PR.
I did a major rewrite to cover presubmit jobs in general, not just those with run_if_changed.
The "blocking presubmits must always run" is enforced by https://github.com/kubernetes/test-infra/pull/33958
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle stale - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle stale - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle stale - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
@BenTheElder: can we get back to this?
Once we agree on the policy, we can proceed with enforcing it in https://github.com/kubernetes/test-infra/pull/33958.
