helm-charts
helm-charts copied to clipboard
Published
20 hours ago •
dragonflyoss
dragonflyoss
pod CreateContainerError when deploy according INSTALL.md
trafficstars
Bug report:
When I tested according to the steps in INSTALL.md, there was an error deploying the NGINX pod
pod events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 43s default-scheduler Successfully assigned default/nginx to kind-control-plane
Normal Pulled 40s kubelet Successfully pulled image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest" in 1.470385199s (1.470388746s including waiting)
Warning Failed 19s kubelet Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error
Normal Pulling 18s (x2 over 41s) kubelet Pulling image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest"
Normal Pulled 17s kubelet Successfully pulled image "ghcr.io/dragonflyoss/image-service/nginx:nydus-latest" in 1.54053078s (1.540535229s including waiting)
kubelet log:
Sep 04 08:42:36 kind-control-plane kubelet[214]: I0904 08:42:36.186657 214 topology_manager.go:200] "Topology Admit Handler"
Sep 04 08:42:36 kind-control-plane kubelet[214]: I0904 08:42:36.324750 214 reconciler.go:238] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b29\" (UniqueName: \"kubernetes.io/projected/3e90bdfa-1f46-4f8c-ad64-e490c7a32095-kube-api-access-57b29\") pod \"nginx\" (UID: \"3e90bdfa-1f46-4f8c-ad64-e490c7a32095\") " pod="default/nginx"
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211022 214 remote_runtime.go:416] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error" podSandboxID="13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c"
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211143 214 kuberuntime_manager.go:919] container &Container{Name:nginx,Image:ghcr.io/dragonflyoss/image-service/nginx:nydus-latest,Command:[sh -c],Args:[tail -f /dev/null],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-57b29,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod nginx_default(3e90bdfa-1f46-4f8c-ad64-e490c7a32095): CreateContainerError: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error
Sep 04 08:43:00 kind-control-plane kubelet[214]: E0904 08:43:00.211181 214 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1614983617: read /var/lib/containerd/tmpmounts/containerd-mount1614983617/etc/passwd: input/output error\"" pod="default/nginx" podUID=3e90bdfa-1f46-4f8c-ad64-e490c7a32095
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203762 214 remote_runtime.go:416] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error" podSandboxID="13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c"
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203865 214 kuberuntime_manager.go:919] container &Container{Name:nginx,Image:ghcr.io/dragonflyoss/image-service/nginx:nydus-latest,Command:[sh -c],Args:[tail -f /dev/null],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-57b29,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod nginx_default(3e90bdfa-1f46-4f8c-ad64-e490c7a32095): CreateContainerError: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error
Sep 04 08:43:23 kind-control-plane kubelet[214]: E0904 08:43:23.203899 214 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3881930381: read /var/lib/containerd/tmpmounts/containerd-mount3881930381/etc/passwd: input/output error\"" pod="default/nginx" podUID=3e90bdfa-1f46-4f8c-ad64-e490c7a32095
nydus-snapshotter pod log:
time="2024-09-04T08:42:15.752400018Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/439]"
time="2024-09-04T08:42:38.152455616Z" level=info msg="[Mounts] snapshot k8s.io/470/13d3d6d05234d011ee1fc0f6ac78473c2afa0b750bd66f7c08eb753659c4b67c ID 468 Kind Active"
time="2024-09-04T08:42:39.704748418Z" level=info msg="Prepares active snapshot k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f, nydusd should start afterwards" key=k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:42:39.704785089Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.704790490Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.704801601Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/469/work upperdir=/var/lib/containerd-nydus/snapshots/469/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:42:39.705652387Z" level=info msg="[Mounts] snapshot k8s.io/471/0fa090b5c0091acff6f6955b49a3c19efa689651f1ece985b3adb304a12da12f ID 469 Kind Active"
time="2024-09-04T08:42:39.705669600Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:42:39.705685702Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/469/work upperdir=/var/lib/containerd-nydus/snapshots/469/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:42:41.706689 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:42:43.707298 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:42:48.707620 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:42:54.208436 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:00.208952 +00:00] ERROR [error/src/error.rs:21] Error:
Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
at storage/src/cache/mod.rs:207
note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:00.208997 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4616 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }
time="2024-09-04T08:43:00.425075289Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/469]"
time="2024-09-04T08:43:02.692829844Z" level=info msg="Prepares active snapshot k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118, nydusd should start afterwards" key=k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118 parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:43:02.692879661Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.692884942Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.692896324Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/470/work upperdir=/var/lib/containerd-nydus/snapshots/470/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:43:02.695391961Z" level=info msg="[Mounts] snapshot k8s.io/472/3a124f67f76a572e18dc780272f44896bb901c998bf5ccf2aa8c21ea5291c118 ID 470 Kind Active"
time="2024-09-04T08:43:02.695420046Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:02.695529268Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/470/work upperdir=/var/lib/containerd-nydus/snapshots/470/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:43:04.698170 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:43:06.698608 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:43:11.698944 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:43:17.199828 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:23.200441 +00:00] ERROR [error/src/error.rs:21] Error:
Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
at storage/src/cache/mod.rs:207
note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:23.200507 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4626 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }
time="2024-09-04T08:43:23.234780304Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/470]"
time="2024-09-04T08:43:26.150099852Z" level=info msg="Prepares active snapshot k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53, nydusd should start afterwards" key=k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53 parent="k8s.io/11/sha256:f7a1174747879fefb6b2dbea73b25fe692b41825ef22f70ee21b1b301fb657ff"
time="2024-09-04T08:43:26.150243962Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.150265785Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.150293529Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/471/work upperdir=/var/lib/containerd-nydus/snapshots/471/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
time="2024-09-04T08:43:26.151552388Z" level=info msg="[Mounts] snapshot k8s.io/473/b0de897205298e4f4cef3730e63f0416440f794cd0453e604c00f0a6b3d3bf53 ID 471 Kind Active"
time="2024-09-04T08:43:26.151618537Z" level=info msg="Nydus remote snapshot 9 is ready"
time="2024-09-04T08:43:26.151743851Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/471/work upperdir=/var/lib/containerd-nydus/snapshots/471/fs lowerdir=/var/lib/containerd-nydus/snapshots/9/mnt]"
[2024-09-04 08:43:28.153610 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 5, compressed offset 7377
[2024-09-04 08:43:30.154184 +00:00] WARN [storage/src/cache/state/blob_state_map.rs:120] Waiting for backend IO expires. chunk index 6, compressed offset 7959
[2024-09-04 08:43:35.154426 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 2
[2024-09-04 08:43:40.655234 +00:00] WARN [storage/src/backend/mod.rs:91] Read from backend failed: Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut }))), retry count 1
[2024-09-04 08:43:46.656648 +00:00] ERROR [error/src/error.rs:21] Error:
Registry(Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ghcr.io")), port: None, path: "/v2/dragonflyoss/image-service/nginx/blobs/sha256:a877ac8e60e55738ac9a27ed567b13843d46e4a71ce044811d0e30de06b68c94", query: None, fragment: None }, source: TimedOut })))
at storage/src/cache/mod.rs:207
note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2024-09-04 08:43:46.656693 +00:00] ERROR [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:1180] fuse: reply error header OutHeader { len: 16, error: -5, unique: 4636 }, error Os { code: 5, kind: Uncategorized, message: "I/O error" }
configs:
kind config:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: dual
nodes:
- role: control-plane
image: kindest/node:v1.23.17
extraPortMappings:
- containerPort: 4001
hostPort: 4001
protocol: TCP
extraMounts:
- hostPath: ./config.toml
containerPath: /etc/containerd/config.toml
- hostPath: /tmp/artifact
containerPath: /tmp/artifact
- hostPath: /dev/fuse
containerPath: /dev/fuse
containerd config:
# explicitly use v2 config format
version = 2
[debug]
level = "debug"
[plugins."io.containerd.grpc.v1.cri".containerd]
# save disk space when using a single snapshotter
discard_unpacked_layers = false
# explicitly use default snapshotter so we can sed it in entrypoint
snapshotter = "nydus"
# explicit default here, as we're configuring it below
default_runtime_name = "runc"
disable_snapshot_annotations = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
# set default runtime handler to v2, which has a per-pod shim
runtime_type = "io.containerd.runc.v2"
# Setup a runtime with the magic name ("test-handler") used for Kubernetes
# runtime class tests ...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri"]
# use fixed sandbox image
sandbox_image = "registry.k8s.io/pause:3.5"
# allow hugepages controller to be missing
# see https://github.com/containerd/cri/pull/1501
tolerate_missing_hugepages_controller = true
# restrict_oom_score_adj needs to be true when running inside UserNS (rootless)
restrict_oom_score_adj = false
# Mirrors use d7y.io instead of docker.io.
# when using docker.io to generate an endpoint, https://index.docker.io will be returned by default.
# https://github.com/containerd/containerd/blob/main/pkg/cri/server/image_pull.go#L456
# https://github.com/containerd/containerd/blob/main/remotes/docker/resolver.go#L121
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."d7y.io"]
endpoint = ["http://127.0.0.1:65001"]
[proxy_plugins]
[proxy_plugins.nydus]
type = "snapshot"
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
nydus helm config:
name: nydus-snapshotter
pullPolicy: Always
hostNetwork: true
resources:
requests:
cpu: "0"
memory: "0"
limits:
cpu: "1"
memory: "1Gi"
dragonfly:
enable: true
mirrorConfig:
- host: "http://127.0.0.1:65001"
auth_through: false
headers:
"X-Dragonfly-Registry": "https://ghcr.io"
ping_url: "http://127.0.0.1:40901/server/ping"
containerRuntime:
initContainerImage: ghcr.io/liubin/toml-cli:v0.0.7
containerd:
enable: true
configFile: "/etc/containerd/config.toml"
Environment:
- Dragonfly version: v2.1.40
- OS: Ubuntu 22.04 LTS（vm）
- Kernel (e.g.
uname -a): 6.8.0-40-generic（vm）
