zos
zos copied to clipboard
Published
20 hours ago •
threefoldtech
threefoldtech
qsfs mounted on /var/lib/docker not working as expected
i used terraform conf file to delpoy a vm with qsfs volume to test.
if qsfs volume is mounted on /var/lib/docker, the docker daemon will show some errors when started and will not be able to save images in the docker directory.
ERRO[2021-10-14T00:31:38.585981823Z] failed to mount overlay: function not implemented storage-driver=overlay2
ERRO[2021-10-14T00:31:38.591254931Z] exec: "fuse-overlayfs": executable file not found in $PATH storage-driver=fuse-overlayfs
ERRO[2021-10-14T00:31:38.605236802Z] AUFS was not found in /proc/filesystems storage-driver=aufs
ERRO[2021-10-14T00:31:38.629026028Z] failed to mount overlay: function not implemented storage-driver=overlay
ERRO[2021-10-14T00:31:38.630029245Z] Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options storage-driver=devicemapper
if I try to download an image, it will fail with input/output error.
vm1:/# docker run -dit --name my-running-app -p 8080:80 httpd:2.4
Unable to find image 'httpd:2.4' locally
2.4: Pulling from library/httpd
b380bbd43752: Pull complete
d0c6942edac3: Pull complete
d027638c026c: Pull complete
17082a2122d5: Pull complete
0b7d42e498cc: Pull complete
Digest: sha256:f03a63735d3653045a3b1f5490367415e9534d8abfe0b21252454dc85ca09800
Status: Downloaded newer image for httpd:2.4
ERRO[2021-10-14T00:35:54.127680460Z] Handler for POST /v1.41/containers/create returned error: symlink libext2fs.so.2.4 /var/lib/docker/vfs/dir/589193b59dec0c966863b4893e9d56255b654bcfb7ac2ebb0c6ce61f44563c56-init/lib/x86_64-linux-gnu/libext2fs.so.2: input/output error
docker: Error response from daemon: symlink libext2fs.so.2.4 /var/lib/docker/vfs/dir/589193b59dec0c966863b4893e9d56255b654bcfb7ac2ebb0c6ce61f44563c56-init/lib/x86_64-linux-gnu/libext2fs.so.2: input/output error.
See 'docker run --help'.
mount a normal volume on the same path /var/lib/docker will not show the errors when docker daemon starting, and it will be able to download images and start containers normally.
docked full starting logs when qsfs mount used
m1:/# INFO[2021-10-14T00:31:37.978119960Z] Starting up
INFO[2021-10-14T00:31:38.065237508Z] libcontainerd: started new containerd process pid=254
INFO[2021-10-14T00:31:38.065892904Z] parsed scheme: "unix" module=grpc
INFO[2021-10-14T00:31:38.066070061Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2021-10-14T00:31:38.066232889Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2021-10-14T00:31:38.066541459Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2021-10-14T00:31:38.219446439Z] starting containerd revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4
INFO[2021-10-14T00:31:38.301253826Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2021-10-14T00:31:38.308883928Z] loading plugin "io.containerd.snapshotter.v1.aufs"... type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.325156116Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"... error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.325313520Z] loading plugin "io.containerd.snapshotter.v1.btrfs"... type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.328964585Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"... error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (virtiofs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.329211452Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
WARN[2021-10-14T00:31:38.329312059Z] failed to load plugin io.containerd.snapshotter.v1.devmapper error="devmapper not configured"
INFO[2021-10-14T00:31:38.329380119Z] loading plugin "io.containerd.snapshotter.v1.native"... type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.335195945Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... type=io.containerd.snapshotter.v1
WARN[2021-10-14T00:31:38.344246104Z] failed to load plugin io.containerd.snapshotter.v1.overlayfs error="/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
INFO[2021-10-14T00:31:38.344647394Z] loading plugin "io.containerd.snapshotter.v1.zfs"... type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.345122161Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"... error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-10-14T00:31:38.345216241Z] loading plugin "io.containerd.metadata.v1.bolt"... type=io.containerd.metadata.v1
WARN[2021-10-14T00:31:38.348455643Z] could not use snapshotter devmapper in metadata plugin error="devmapper not configured"
WARN[2021-10-14T00:31:38.348576350Z] could not use snapshotter overlayfs in metadata plugin error="/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
INFO[2021-10-14T00:31:38.348804530Z] metadata content store policy set policy=shared
INFO[2021-10-14T00:31:38.361313134Z] loading plugin "io.containerd.differ.v1.walking"... type=io.containerd.differ.v1
INFO[2021-10-14T00:31:38.361452231Z] loading plugin "io.containerd.gc.v1.scheduler"... type=io.containerd.gc.v1
INFO[2021-10-14T00:31:38.361571641Z] loading plugin "io.containerd.service.v1.introspection-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.361730751Z] loading plugin "io.containerd.service.v1.containers-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.361809291Z] loading plugin "io.containerd.service.v1.content-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.361895731Z] loading plugin "io.containerd.service.v1.diff-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.361961531Z] loading plugin "io.containerd.service.v1.images-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.362018181Z] loading plugin "io.containerd.service.v1.leases-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.362072991Z] loading plugin "io.containerd.service.v1.namespaces-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.362127568Z] loading plugin "io.containerd.service.v1.snapshots-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.362180964Z] loading plugin "io.containerd.runtime.v1.linux"... type=io.containerd.runtime.v1
INFO[2021-10-14T00:31:38.382948210Z] loading plugin "io.containerd.runtime.v2.task"... type=io.containerd.runtime.v2
INFO[2021-10-14T00:31:38.406024901Z] loading plugin "io.containerd.monitor.v1.cgroups"... type=io.containerd.monitor.v1
INFO[2021-10-14T00:31:38.409472950Z] loading plugin "io.containerd.service.v1.tasks-service"... type=io.containerd.service.v1
INFO[2021-10-14T00:31:38.410086220Z] loading plugin "io.containerd.internal.v1.restart"... type=io.containerd.internal.v1
INFO[2021-10-14T00:31:38.410795794Z] loading plugin "io.containerd.grpc.v1.containers"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411055221Z] loading plugin "io.containerd.grpc.v1.content"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411138799Z] loading plugin "io.containerd.grpc.v1.diff"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411229514Z] loading plugin "io.containerd.grpc.v1.events"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411314899Z] loading plugin "io.containerd.grpc.v1.healthcheck"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411399314Z] loading plugin "io.containerd.grpc.v1.images"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411494131Z] loading plugin "io.containerd.grpc.v1.leases"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411549364Z] loading plugin "io.containerd.grpc.v1.namespaces"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.411732984Z] loading plugin "io.containerd.internal.v1.opt"... type=io.containerd.internal.v1
INFO[2021-10-14T00:31:38.417082177Z] loading plugin "io.containerd.grpc.v1.snapshots"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.417253672Z] loading plugin "io.containerd.grpc.v1.tasks"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.417309625Z] loading plugin "io.containerd.grpc.v1.version"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.417366208Z] loading plugin "io.containerd.grpc.v1.introspection"... type=io.containerd.grpc.v1
INFO[2021-10-14T00:31:38.437524982Z] serving... address=/var/run/docker/containerd/containerd-debug.sock
INFO[2021-10-14T00:31:38.458576378Z] serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2021-10-14T00:31:38.474999714Z] serving... address=/var/run/docker/containerd/containerd.sock
INFO[2021-10-14T00:31:38.475581007Z] containerd successfully booted in 0.280236s
INFO[2021-10-14T00:31:38.515874160Z] parsed scheme: "unix" module=grpc
INFO[2021-10-14T00:31:38.516029683Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2021-10-14T00:31:38.516098630Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2021-10-14T00:31:38.516131707Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2021-10-14T00:31:38.527000384Z] parsed scheme: "unix" module=grpc
INFO[2021-10-14T00:31:38.527083547Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2021-10-14T00:31:38.527199660Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2021-10-14T00:31:38.527239430Z] ClientConn switching balancer to "pick_first" module=grpc
ERRO[2021-10-14T00:31:38.585981823Z] failed to mount overlay: function not implemented storage-driver=overlay2
ERRO[2021-10-14T00:31:38.591254931Z] exec: "fuse-overlayfs": executable file not found in $PATH storage-driver=fuse-overlayfs
ERRO[2021-10-14T00:31:38.605236802Z] AUFS was not found in /proc/filesystems storage-driver=aufs
ERRO[2021-10-14T00:31:38.629026028Z] failed to mount overlay: function not implemented storage-driver=overlay
ERRO[2021-10-14T00:31:38.630029245Z] Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options storage-driver=devicemapper
WARN[2021-10-14T00:31:38.722360290Z] Your kernel does not support CPU realtime scheduler
WARN[2021-10-14T00:31:38.722430533Z] Your kernel does not support cgroup blkio weight
WARN[2021-10-14T00:31:38.722453983Z] Your kernel does not support cgroup blkio weight_device
INFO[2021-10-14T00:31:38.722933277Z] Loading containers: start.
WARN[2021-10-14T00:31:38.753576571Z] Failed to find ip6tables: exec: "ip6tables": executable file not found in $PATH
INFO[2021-10-14T00:31:39.185606140Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2021-10-14T00:31:39.453686274Z] Loading containers: done.
INFO[2021-10-14T00:31:39.525360626Z] Docker daemon commit=46229ca1d815cfd4b50eb377ac75ad8300e13a85 graphdriver(s)=vfs version=20.10.3
INFO[2021-10-14T00:31:39.538124173Z] Daemon has completed initialization
INFO[2021-10-14T00:31:39.664916147Z] API listen on /var/run/docker.sock
This is (from experience) a fuse filesystem issue. the zdb-fs is probably missing the statfs call implementation. and then the overlay filesystem fails to use it.
@maxux
statfs is implemented.
Docker use special driver for each filesystem. It make sens for me that it doesn't support fuse stuff. Why would you do that anyway ?!
It could maybe works by forcing some legacy driver ? Anyway this is not a zdbfs issue but docker issue.
@maxux latest update that this might be related to zdbfs not working as upper layer in an overlay mount. So can we link this issue here with update if this gonna be fixed or not ?
https://github.com/threefoldtech/0-db-fs/issues/29 Again it's not zdbfs issue, but any fuse fs not supporting this.