test-infra
test-infra copied to clipboard
convert some jobs to Ginkgo --label-filter
Why we need this change:
- https://github.com/kubernetes/test-infra/pull/32648
- https://github.com/kubernetes/kubernetes/pull/125452
- Ginkgo Proposal - Label Sets
How to do it
-
Some tips for figuring out how to replace
--focus/skip
with--label-filter
:-
go test -v ./test/e2e -args -list-tests
(same fore2e_node
) shows all tests. - Note that not all tags in
[]
are also labels. Only those listed bygo test -v ./test/e2e -args -list-labels
are proper labels. If you find that you need to filter by something that isn't a label yet, then replace the inline text withWithLabel
. -
_output/bin/ginkgo --dry-run [--focus/skip/label-filter=...] ./test/e2e
can be used to compare how many and, when adding-v
, which tests would run. Note that statistics for actual runs may differ when tests ask to be skipped when invoked. (install ginkgo viamake ginkgo
)
-
-
If a job has a "canary" version, update that version first and try it out.
-
When converting some real jobs:
- Create a separate tracking issue with a link to the jobs' testgrid entry.
- Convert the job, linking to that issue with
Related-to: ...
(notFixes: ...
!). - Verify that the job still works as intended, then close the issue.
-
After converting a pre-submit job successfully, do the same for the corresponding periodic job.
-
Some best practices:
- Replace focus/skip completely with a single label-filter.
- Include
!Flaky
even if you currently don't have flaky tests. Having it in the jobs is convenient when some tests turns out to be problematic and needs to be disable temporarily. - Exclude tests with special requirements with
Feature: isEmpty
. -
Feature: containsAny <my feature> && Feature: isSubsetOf <my feature>
runs tests with have the<my feature>
label and no other unknown features. Use this in dedicated jobs for that feature. Note that e2e_node usesNodeFeature
for the same purpose. - Ginkgo can run serial jobs together with parallel ones in a single invocation by running the serial jobs (and only those) sequentially, so it's not absolutely required to use
!Serial
when runningginkgo -p
and there is no need for separate serial/parallel jobs. - Beware that serial tests can affect the overall runtime. It might be useful to skip with
!Slow
and!Serial
in pre-submits and only run the skipped tests in periodic jobs.
What should be cleaned up or changed:
- [ ] sig-api-machinery
- [ ] sig-apps
- [ ] sig-arch
- [ ] sig-auth
- [ ] sig-autoscaling
- [ ] sig-cli
- [ ] sig-cloud-provider
- [ ] sig-cluster-lifecycle
- [ ] sig-instrumentation
- [ ] sig-k8s-infra
- [ ] sig-network
- [ ] sig-node
- [ ] sig-release
- [ ] sig-scalability
- [ ] sig-scheduling
- [ ] sig-security
- [ ] https://github.com/kubernetes/test-infra/issues/33152 @carlory
- [ ] sig-testing
- [ ] https://github.com/kubernetes/test-infra/pull/32898 (we also need to clear skip)
- [ ] https://github.com/kubernetes/test-infra/pull/32867 (but reverted by https://github.com/kubernetes/test-infra/pull/33131 due to https://github.com/kubernetes/kubernetes/issues/126401)
- [ ] sig-windows