sonobuoy
sonobuoy copied to clipboard
Published
20 hours ago •
vmware-tanzu
vmware-tanzu
Sonobuoy run with --e2e-repo flage
What steps did you take and what happened:
I executed the sonobuoy run command with --e2e-repo flag instead the --e2e-repo-config. I tried to avoid creating such custom-repo config file as I use the same repo for all images.
sonobuoy run \
--sonobuoy-image bcmt-registry:5000/sonobuoy:v0.57.2 \
--kube-conformance-image bcmt-registry:5000/conformance:v1.27.16 \
--systemd-logs-image bcmt-registry:5000/systemd-logs:v0.4 \
--plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=is_control,is_edge" \
--e2e-repo bcmt-registry:5000 \
--mode=certified-conformance
Unfortunately it failed, immediately after started.
POD
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
sonobuoy sonobuoy 1/1 Running 0 8s
sonobuoy sonobuoy-e2e-job-ffad56eeb013477d 1/2 Error 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-49w7q 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-9l7c7 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-chvbd 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-cnd4j 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-mwm4s 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-rh7ll 2/2 Running 0 6s
sonobuoy sonobuoy-systemd-logs-daemon-set-7a6e8f671c434e9a-rtqp2 2/2 Running 0 6s
LOG E2E
kubectl logs -n sonobuoy sonobuoy-e2e-job-ffad56eeb013477d
Defaulted container "e2e" out of: e2e, sonobuoy-worker
2024/10/14 09:13:44 warn: calling test with e2e.test is deprecated and will be removed in 1.25, please rely on container manifest to invoke executable
sep is " " args are "--non-blocking-taints=is_control,is_edge"split [--non-blocking-taints=is_control,is_edge]
2024/10/14 09:13:44 Running command:
Command env: []
Run from directory:
Executable path: /usr/local/bin/ginkgo
Args (comma-delimited): /usr/local/bin/ginkgo,--focus=\[Conformance\],--skip=,--noColor=true,--timeout=24h,/usr/local/bin/e2e.test,--,--disable-log-dump,--repo-root=/kubernetes,--provider=local,--report-dir=/tmp/sonobuoy/results,--kubeconfig=,--non-blocking-taints=is_control,is_edge
2024/10/14 09:13:44 Now listening for interrupts
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
k8s.io/kubernetes/test/utils/image.getRepositoryMappedConfig(0xc, {{0x6d5796c, 0x1f}, {0x6d34336, 0x1a}, {0x6ccc9e5, 0x4}}, {0xc00007a09f?, 0x7558b20?})
k8s.io/kubernetes/test/utils/image/manifest.go:329 +0x54a
k8s.io/kubernetes/test/utils/image.GetMappedImageConfigs(0xc000827950, {0xc00007a09f, 0x12})
k8s.io/kubernetes/test/utils/image/manifest.go:312 +0x174
k8s.io/kubernetes/test/utils/image.initImageConfigs({{0x6d6ef42, 0x22}, {0x6d5796c, 0x1f}, {0x6d3b0f2, 0x1b}, {0x6d5798b, 0x1f}, {0x6cf20ed, 0xf}, ...})
k8s.io/kubernetes/test/utils/image/manifest.go:289 +0x22f2
k8s.io/kubernetes/test/utils/image.readRepoList({0x0, 0x0})
k8s.io/kubernetes/test/utils/image/manifest.go:79 +0xe5
k8s.io/kubernetes/test/utils/image.init()
k8s.io/kubernetes/test/utils/image/manifest.go:146 +0xf8
Ginkgo ran 1 suite in 190.441461ms
Test Suite Failed
You're using deprecated Ginkgo functionality:
=============================================
--noColor is deprecated, use --no-color instead
Learn more at: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#changed-command-line-flags
To silence deprecations that can be silenced set the following environment variable:
ACK_GINKGO_DEPRECATIONS=2.9.1
2024/10/14 09:13:44 Saving results at /tmp/sonobuoy/results
2024/10/14 09:13:44 running command: exit status 1
LOG SONOBUOY
kubectl logs -n sonobuoy sonobuoy
time="2024-10-14T09:13:43Z" level=info msg="Scanning plugins in ./plugins.d (pwd: /)"
time="2024-10-14T09:13:43Z" level=info msg="Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)"
time="2024-10-14T09:13:43Z" level=info msg="Directory (/etc/sonobuoy/plugins.d) does not exist"
time="2024-10-14T09:13:43Z" level=info msg="Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)"
time="2024-10-14T09:13:43Z" level=info msg="Directory (~/sonobuoy/plugins.d) does not exist"
time="2024-10-14T09:13:43Z" level=info msg="Starting server Expected Results: [{global e2e 0} {cbis-sut1-control-01.cluster.local systemd-logs 0} {cbis-sut1-control-02.cluster.local systemd-logs 0} {cbis-sut1-control-03.cluster.local systemd-logs 0} {cbis-sut1-edge-01.cluster.local systemd-logs 0} {cbis-sut1-edge-02.cluster.local systemd-logs 0} {cbis-sut1-worker-01.cluster.local systemd-logs 0} {cbis-sut1-worker-02.cluster.local systemd-logs 0}]"
time="2024-10-14T09:13:43Z" level=info msg="Starting annotation update routine"
time="2024-10-14T09:13:43Z" level=info msg="Starting aggregation server" address=0.0.0.0 port=8080
time="2024-10-14T09:13:43Z" level=info msg="Received plugin launch order:"
time="2024-10-14T09:13:43Z" level=info msg="e2e: 0"
time="2024-10-14T09:13:43Z" level=info msg="systemd-logs: 0"
time="2024-10-14T09:13:43Z" level=info msg="Running plugin" plugin=e2e
time="2024-10-14T09:13:43Z" level=info msg="Running plugin" plugin=systemd-logs
time="2024-10-14T09:13:43Z" level=info msg="Launching plugin e2e with order 0"
time="2024-10-14T09:13:43Z" level=info msg="Launching plugin systemd-logs with order 0"
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-edge-02.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-edge-02.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-worker-02.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-worker-02.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-edge-01.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-edge-01.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-control-01.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-control-01.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-control-03.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-control-03.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-worker-01.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-worker-01.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[systemd-logs]" method=PUT node=cbis-sut1-control-02.cluster.local plugin_name=systemd-logs url=/api/v1/results/by-node/cbis-sut1-control-02.cluster.local/systemd-logs
time="2024-10-14T09:13:50Z" level=info msg="received request" client_cert="[e2e]" method=PUT plugin_name=e2e url=/api/v1/results/global/e2e
time="2024-10-14T09:13:51Z" level=info msg="All plugins have completed, status has been updated"
time="2024-10-14T09:13:53Z" level=info msg="Shutting down aggregation server"
time="2024-10-14T09:13:53Z" level=info msg="Resources is not set explicitly implying query all resources, but skipping secrets for safety. Specify the value explicitly in Resources to gather this data."
time="2024-10-14T09:13:53Z" level=info msg="Collecting Node Configuration and Health..."
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-control-01.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-control-01.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-control-02.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-control-02.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-control-03.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-control-03.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-edge-01.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-edge-01.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-edge-02.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-edge-02.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-worker-01.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-worker-01.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Creating host results for cbis-sut1-worker-02.cluster.local under /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/hosts/cbis-sut1-worker-02.cluster.local\n"
time="2024-10-14T09:13:53Z" level=info msg="Running cluster queries"
W1014 09:13:53.510637 1 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
time="2024-10-14T09:13:54Z" level=info msg="Querying server version and API Groups"
time="2024-10-14T09:13:54Z" level=info msg="Filtering namespaces based on the following regex:.*"
time="2024-10-14T09:13:54Z" level=info msg="Namespace cosign-system Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace default Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace gatekeeper-system Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace kube-node-lease Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace kube-public Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace kube-system Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace ncms Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace node-feature-discovery Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Namespace sonobuoy Matched=true"
time="2024-10-14T09:13:54Z" level=info msg="Running ns query (cosign-system)"
time="2024-10-14T09:13:56Z" level=info msg="Running ns query (default)"
time="2024-10-14T09:13:59Z" level=info msg="Running ns query (gatekeeper-system)"
time="2024-10-14T09:14:02Z" level=info msg="Running ns query (kube-node-lease)"
time="2024-10-14T09:14:05Z" level=info msg="Running ns query (kube-public)"
time="2024-10-14T09:14:08Z" level=info msg="Running ns query (kube-system)"
time="2024-10-14T09:14:10Z" level=info msg="Running ns query (ncms)"
time="2024-10-14T09:14:13Z" level=info msg="Running ns query (node-feature-discovery)"
time="2024-10-14T09:14:16Z" level=info msg="Running ns query (sonobuoy)"
time="2024-10-14T09:14:19Z" level=info msg="Namespace cosign-system Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace default Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace gatekeeper-system Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace kube-node-lease Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace kube-public Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace kube-system Matched=true"
time="2024-10-14T09:14:19Z" level=info msg="Namespace ncms Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace node-feature-discovery Matched=false"
time="2024-10-14T09:14:19Z" level=info msg="Namespace sonobuoy Matched=true"
time="2024-10-14T09:14:19Z" level=info msg="querying pod logs under namespace kube-system"
time="2024-10-14T09:14:19Z" level=info msg="Collecting Pod Logs by namespace (kube-system)"
time="2024-10-14T09:14:22Z" level=info msg="querying pod logs under namespace sonobuoy"
time="2024-10-14T09:14:22Z" level=info msg="Collecting Pod Logs by namespace (sonobuoy)"
time="2024-10-14T09:14:22Z" level=info msg="Collecting Pod Logs by FieldSelectors: []"
time="2024-10-14T09:14:22Z" level=info msg="recording query times at /tmp/sonobuoy/f1f670de-b70f-47ce-954f-fbe5a68fd6e5/meta/query-time.json"
time="2024-10-14T09:14:22Z" level=info msg="Log lines after this point will not appear in the downloaded tarball."
time="2024-10-14T09:14:22Z" level=info msg="Invoking plugin cleanup" plugin=e2e
time="2024-10-14T09:14:22Z" level=info msg="Invoking plugin cleanup" plugin=systemd-logs
time="2024-10-14T09:14:25Z" level=info msg="Results available at /tmp/sonobuoy/202410140913_sonobuoy_f1f670de-b70f-47ce-954f-fbe5a68fd6e5.tar.gz"
time="2024-10-14T09:14:25Z" level=info msg="no-exit was specified, sonobuoy is now blocking"
STATUS
sonobuoy status
PLUGIN STATUS RESULT COUNT PROGRESS
e2e complete unknown 1
systemd-logs complete passed 7
Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
What did you expect to happen:
I expected it will work, like it worked properly for image pull.
sonobuoy images pull
sonobuoy images push --custom-registry bcmt-registry:5000 --e2e-repo bcmt-registry:5000
# so I didn't have to create this config file unnecessarily (as I use the same repo for all images)
# sonobuoy images push --custom-registry bcmt-registry:5000 --e2e-repo-config ~/p0t3st-sonobuoy/p0t3st-custom-repo-config.yaml
It seems, I can't avoid creating such custom-repo config file. If that's the case, then I don't really understand the purpose of this --e2e-repo flag because even if I can use it during image pull, tag, push, at the end when I run the test I still need to create and use this custom-repo file with --e2e-repo-config flag, like you see below.
sonobuoy run \
--sonobuoy-image bcmt-registry:5000/sonobuoy:v0.57.2 \
--kube-conformance-image bcmt-registry:5000/conformance:v1.27.16 \
--systemd-logs-image bcmt-registry:5000/systemd-logs:v0.4 \
--plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=is_control,is_edge" \
--e2e-repo-config ~/p0t3st-sonobuoy/p0t3st-custom-repo-config.yaml \
--mode=certified-conformance
Anything else you would like to add:
According to the help the --e2e-repo is supported
sonobuoy run --help
...
--e2e-repo envModifier Specify a registry to use as the default for pulling Kubernetes test images. Same as providing --e2e-repo-config but specifying the same repo repeatedly.
--e2e-repo-config yaml-filepath Specify a yaml file acting as KUBE_TEST_REPO_LIST, overriding registries for test images.
...
Environment:
- Sonobuoy version:
v0.57.2 - Kubernetes version:
v1.27.16 - Cloud provider or hardware configuration:
OpenStack - OS (e.g. from
/etc/os-release):Red Hat Enterprise Linux 8.10 (Ootpa)
