kubetest2-ginko tester overwrites local e2e.test

[mattcary]

Expected behavior: kubetest2 builds and uses a local e2e.test Actual behavior: kubetest2 builds a local e2e.test, but uses a different one.

I'm trying to test out a new e2e.test config flag (https://github.com/kubernetes/kubernetes/pull/111481). I'm running the following kubetest2 command, inspired by the cloud-provider-gcp prow job:

GOPATH=/clank/go/src kubetest2 gce -v 6  --gcp-project mattcary-e2e-test --legacy-mode --repo-root /clank/go/src/k8s.io/kubernetes --build --up --down --test=ginkgo --node-size n1-standard-4 --master-size n1-standard-8 -- --parallel=30 --test-args='--minStartupPods=8 --ginkgo.flakeAttempts=3 --enabled-volume-drivers=gcepd' --skip-regex='\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]' --focus-regex='\[Driver:.gcepd\]'

This is using kubetest2 at commit 2aac35a from 13 July.

The command above correctly builds e2e.test and puts it into a local runfiles. While the gce cluster is coming up, I can confirm that the e2e.test has my new flag.

However, somewhere around here in the logs it's replaced by some version pulled down from somewhere, instead of my local build:

I0728 15:11:35.330046  345616 ginkgo.go:121] Using kubeconfig at /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubetest2-kubeconfig
I0728 15:11:37.148140  345616 package.go:199] Found existing tar at /usr/local/google/home/mattcary/.cache/kubernetes-test-linux-amd64.tar.gz
I0728 15:11:39.380846  345616 package.go:202] Validated hash for existing tar at /usr/local/google/home/mattcary/.cache/kubernetes-test-linux-amd64.tar.gz
I0728 15:11:41.857031  345616 package.go:165] Found existing kubectl at /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubectl
W0728 15:11:43.508631  345616 package.go:171] sha256 does not match
Copying gs://kubernetes-release/release/v1.25.0-alpha.3/bin/linux/amd64/kubectl...
\ [1 files][ 42.0 MiB/ 42.0 MiB]                                                
Operation completed over 1 objects/42.0 MiB.                                     
I0728 15:11:46.098135  345616 ginkgo.go:91] Running ginkgo test as /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/ginkgo [--nodes=30 /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/e2e.test -- --kubeconfig=/clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubetest2-kubeconfig --kubectl-path=/clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubectl --ginkgo.flakeAttempts=1 --ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\] --ginkgo.focus=\[Driver:.gcepd\] --report-dir=/clank/go/src/k8s.io/kubernetes/_artifacts --test.timeout=24h --minStartupPods=8 --ginkgo.flakeAttempts=3 --enabled-volume-drivers=gcepd]

That is, at the log message at 15:11:35 the correct e2e.test is in _rundir/af0111fff*, and by the log message at 15:11:26 its been replaced by a different one.

I've looked through the kubetest2 code and wonder if the problem is here: https://github.com/kubernetes-sigs/kubetest2/blob/master/pkg/testers/ginkgo/package.go#L105, although I'm not really sure what's going on.

[BenTheElder]

Hmm I think you need another flag https://github.com/kubernetes-sigs/kubetest2/blob/c76fb417aa01874c79b5adf703f0f651897c8a9d/pkg/testers/ginkgo/ginkgo.go#L123 which seems like a bad default

[mattcary]

Oh god and since this a gingko flag and the build flag is a deployer flag or whatever there's no easy way to coordinate the flag or give any kind of useful error message or help to the user.

Infinite sadness.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.