test-infra Enable go race detector for unit tests

It was enabled in https://github.com/kubernetes/test-infra/pull/29917 but had to be reverted in https://github.com/kubernetes/test-infra/pull/29917 because it uncovered new tests with races.

Opening the issue to not forget we must fix all races and enable the race detector in presubmits

Jun 23 '23 07:06 aojea

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

/sig <group-name>
/wg <group-name>
/committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Jun 23 '23 07:06 k8s-ci-robot

/assign @listx

Jun 23 '23 07:06 aojea

Link to some of the races that were detected (thanks @pohly for the example): https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/test-infra/29906/pull-test-infra-unit-test/1672049228827856896/build-log.txt. We can start by fixing these first.

Jun 23 '23 16:06 listx

We now have https://testgrid.k8s.io/presubmits-test-infra#unit-test-race-detector-nonblocking&width=20 to look at. So far it looks like the prow/external-plugins/cherrypicker TestCherryPickOfCherryPickPRV2 test is failing, but I'm not sure how to fix it. I don't think there's an explicit race that's been detected, but the test seems to become flaky when we turn on the race detector.

Jun 29 '23 00:06 listx

race detector has a considerable penalty in performance, I would check if the test is making some assumptions on time and is failing because it is too slow now

Jun 29 '23 08:06 aojea

Any updates on this? I saw several consistent failure in different PR https://github.com/kubernetes/test-infra/pull/30308 https://github.com/kubernetes/test-infra/pull/30269 https://github.com/kubernetes/test-infra/pull/30325

Ok it's non-blocking.

Aug 08 '23 20:08 lubronzhan

Yeah I'm now thinking that one way to fix this would be to modify the test jobs so that we exclude certain tests from being run together with the race detector (because trying to fix the race-detector-unfriendly jobs is rather time consuming and I don't think anyone has the bandwidth for it).

Aug 08 '23 21:08 listx

Yeah I'm now thinking that one way to fix this would be to modify the test jobs so that we exclude certain tests

I think is a good idea ... but how do you plan to implement that?

Aug 09 '23 15:08 aojea

I've implemented something like that for Kubernetes integration tests: https://github.com/kubernetes/kubernetes/pull/116980/commits/f2a237a8039e3a96eca969e526dda15e23844048

Aug 09 '23 16:08 pohly

No idea whether that is even remotely applicable to how test-infra runs unit tests...

Aug 09 '23 16:08 pohly

I think it is applicable. I haven't looked too deeply here but we should be able to basically break out the unit test invocation into 2 separate invocations. That seems to be what you're doing in the link you provided (thanks for the reference!).

Aug 09 '23 21:08 listx

/cc @airbornepony

Sep 22 '23 17:09 listx

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Jan 28 '24 23:01 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Feb 27 '24 23:02 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Mar 29 '24 00:03 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Mar 29 '24 00:03 k8s-ci-robot