kubernetes-sigs
feat: add vcluster e2e cluster provider
addition of vcluster provider; interested to see if the CI fails. vcluster needs a "host" cluster to launch into, so I've followed an existing example where a real cluster or kind cluster are utilized as necessary: https://github.com/kubernetes-sigs/e2e-framework/blob/9206e6f6082907a1c77750b7cac0fc44ff4dac02/examples/real_cluster/main_test.go#L39
What type of PR is this?
/kind feature
What this PR does / why we need it:
This PR enables a new E2EClusterProvider for vcluster based clusters.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?
NONE
Additional documentation e.g., Usage docs, etc.:
NA
Hi @crandles. Thanks for your PR.
I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@crandles this is awesome and thanks for the contribution. At first glance it looks Okay, but I will ask @harshanarayana to take a look as well since he's the E2E provider expert 😉
golangci appears to be passing now for thew new code I've added, though errirng for some existing LOC:
third_party/helm/helm.go:240:9: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
err = fmt.Errorf(missingHelm)
^
third_party/flux/flux_setup.go:47:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
^
third_party/flux/flux_setup.go:61:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
^
pkg/env/env.go:229:11: printf: non-constant format string in call to (*testing.common).Skipf (govet)
t.Skipf(message)
^
pkg/env/env.go:513:22: printf: non-constant format string in call to (*testing.common).Skipf (govet)
internalT.Skipf(message)
Assuming that won't block this PR.
@harshanarayana and @cpanato any idea why golangci failing on previously working code ?
errors were primarily for:
SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
It's possible that I'm using a different toolchain version, but as the warning seemed fair, I've fixed it via https://github.com/kubernetes-sigs/e2e-framework/pull/450/commits/cf9cbf28dfc7a407ed901b2da26b31ef4aabcc35
make verify now passes locally.
@harshanarayana @ShwethaKumbla @cpanato PTAL I think we need a second reviewer.
Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)
Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)
@crandles
Helm and flux are installed slightly in a different way, https://github.com/kubernetes-sigs/e2e-framework/blob/777cbcd59fbbe572b71dde89c8f7b0b1a5ddc4a7/hack/install-helm.sh
test: install-helm install-flux ## Runs golang unit tests
./hack/test-go.sh
But you are right about ko or kwok though
Is there a recommended way to simulate the prow test runner locally?
This looks relevant: https://docs.prow.k8s.io/docs/build-test-update/#running-a-prowjob-locally (and this job), I'll try to follow it.
Trying this out so far, not quite working yet:
# new makefile target
run-pr-tests:
mkdir -p ./build ./build/out ./build/node_dir
curl -s -o build/e2e-framework-presubmits.yaml https://raw.githubusercontent.com/kubernetes/test-infra/master/config/jobs/kubernetes-sigs/e2e-framework/e2e-framework-presubmits.yaml
curl -s -o build/pj-on-kind.sh https://raw.githubusercontent.com/kubernetes-sigs/prow/refs/heads/main/pkg/pj-on-kind.sh
curl -s -o build/prow-config.yaml https://raw.githubusercontent.com/kubernetes/test-infra/refs/heads/master/config/prow/config.yaml
NODE_DIR=$(realpath build/node_dir) OUT_DIR=$(realpath build/out) CONFIG_PATH=$(realpath build/prow-config.yaml) JOB_CONFIG_PATH=$(realpath build/e2e-framework-presubmits.yaml) bash ./build/pj-on-kind.sh pull-e2e-framework-test
Output
latest: Pulling from k8s-prow/mkpj
Digest: sha256:6e71c2d9f675bce23136443cd8093fad59ed3344325d305963e89eadb7d3b6d5
Status: Image is up to date for gcr.io/k8s-prow/mkpj:latest
gcr.io/k8s-prow/mkpj:latest
What's next:
View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpj:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
time="2024-11-13T04:38:37Z" level=warning msg="empty -github-token-path, will use anonymous github client"
time="2024-11-13T04:38:37Z" level=info msg="Throttle(0, 0, [])" client=github
PR Number: 450
time="2024-11-13T04:39:20Z" level=info msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450)" client=github
time="2024-11-13T04:39:20Z" level=debug msg="Using GitHub REST API Version: 2022-11-28" client=github
time="2024-11-13T04:39:20Z" level=debug msg="Used token for request" cache-mode= client=github method=GET path=/repos/kubernetes-sigs/e2e-framework/pulls/450 throttled=true
time="2024-11-13T04:39:20Z" level=debug msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450) finished" client=github duration=419.387292ms
latest: Pulling from k8s-prow/mkpod
Digest: sha256:1a95d701f6cafae2c004088e47f15efc0b8e5b8d8fb6a23a2e2ccd94471eda46
Status: Image is up to date for gcr.io/k8s-prow/mkpod:latest
gcr.io/k8s-prow/mkpod:latest
What's next:
View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpod:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
time="2024-11-13T04:39:22Z" level=info msg="Generated build-id for job." build-id=1856557443136360448
time="2024-11-13T04:39:22Z" level=info msg="Pod-utils configured for local mode. Instead of uploading to GCS, files will be copied to an output dir on the node." out-dir=/Users/c/code/e2e-framework/build/out/pull-e2e-framework-test/1856557443136360448
For each of the following volumes specify one of:
- 'empty' to use an emptyDir;
- a path on the host to use hostPath;
- '' (nothing) to use the existing volume source and assume it is available in the cluster
Volume "service": empty
Applying pod to the mkpod cluster. Configure kubectl for the mkpod cluster with:
> export KUBECONFIG='/Users/c/.kube/kind-config-mkpod'
NAME READY STATUS RESTARTS AGE
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:0/3 0 0s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:0/3 0 4s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:1/3 0 7s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:2/3 0 10s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:2/3 0 12s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 PodInitializing 0 13s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0 2/2 Running 0 70s
---
% kubectl logs -f aed6b1e8-6ada-4b27-90ad-c7387846b7b0
Defaulted container "test" out of: test, sidecar, clonerefs (init), initupload (init), place-entrypoint (init)
Docker in Docker enabled, initializing...
================================================================================
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
Starting Docker: docker.
Waiting for docker to be ready, sleeping for 1 seconds.
================================================================================
Done setting up docker in docker.
ERROR: (gcloud.auth.activate-service-account) Unable to read file [/etc/service-account/service-account.json]: [Errno 2] No such file or directory: '/etc/service-account/service-account.json'
+ WRAPPED_COMMAND_PID=271
+ wait 271
+ make test
./hack/install-helm.sh
Downloading https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz
Verifying checksum... Done.
...
./hack/test-go.sh
go: downloading sigs.k8s.io/controller-runtime v0.19.1
go: downloading k8s.io/klog/v2 v2.130.1
go: downloading k8s.io/client-go v0.31.2
go: downloading k8s.io/apimachinery v0.31.2
...
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown.
...
found/resolved a couple of issues:
- I was mistaken on the vcluster installation path, it is vclusterctl not vcluster
- Apparently I've been falling back to an alternatively installed vcluster in my CI
- there was a flaw in the version defaulting logic
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: cpanato, crandles, vladimirvivien
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [cpanato,vladimirvivien]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
Thanks for the reviews!
I will try to rebase next time, and I may follow up with an issue about how to run the pr tests locally. I need to spend more time looking at the pj on kind script.