test-infra
test-infra copied to clipboard
kubernetes
Containerd E2E test fails with failed to load cni during init
What happened:
Two unrelated docker based e2e tests were moved to containerd in separate PRs (https://github.com/kubernetes/test-infra/pull/25254, https://github.com/kubernetes/test-infra/pull/23243).
Job links :
https://k8s-testgrid.appspot.com/sig-node-kubelet#kubelet-gce-e2e-lock-contention https://k8s-testgrid.appspot.com/sig-node-containerd#node-kubelet-containerd-performance-test
Both the tests are continously failing with the same error mesage:
log snippet from containerd
Feb 22 14:31:42 tmp-node-e2e-1026ef4c-cos-89-16108-604-11 containerd[339]: time="2022-02-22T14:31:42.088924479Z"
level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods"
error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
..
..
..
Feb 22 14:38:41 tmp-node-e2e-1026ef4c-cos-89-16108-604-11 containerd[339]: time="2022-02-22T14:38:41.086102911Z"
level=info msg="No cni config template is specified, wait for other system components to drop the config."
kubelet.log also has the same messages regarding cni plugin not ready
I0222 14:38:54.019413 2109 kubelet.go:2323] "Container runtime status" status="Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
E0222 14:38:54.019441 2109 kubelet.go:2326] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
What you expected to happen:
The test to run successfully.
/cc: @Namanl2001 @adisky @SergeyKanzhelev
what's the diff in test definition with one of the working jobs?
What do you mean by test definition ?
one of the tests mentioned here doesn't have
--container-runtime-endpoint=unix:///run/containerd/containerd.sock
--container-runtime-process-name=/usr/bin/containerd
--container-runtime-pid-file=
but the other one does yet it fails and it matches the one of the successful tests.
Same is with the name of test: one doesn't contain the prefix -containerd-, but other does.
@ipochi the CNI init error is resolved now, but tests are still failing and now both the tests have different errors tracking node performance test here https://github.com/kubernetes/test-infra/issues/25430 tracking lock contention tests here https://github.com/kubernetes/kubernetes/issues/108348
@ipochi the CNI init error is resolved now, but tests are still failing and now both the tests have different errors tracking node performance test here https://github.com/kubernetes/test-infra/issues/25430 tracking lock contention tests here https://github.com/kubernetes/kubernetes/issues/108348
Hi @adisky
Looking into it.
https://github.com/kubernetes/kubernetes/issues/108348#issuecomment-1058933790
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
kubelet-gce-e2e-lock-contention is green now. node-kubelet-containerd-performance-test failed but not for cni not ready
Not sure if it is related with https://github.com/kubernetes/test-infra/issues/25430
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs.
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle frozen
This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Deprioritize it with
/priority important-longtermor/priority backlog - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
@SergeyKanzhelev: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
