docker-volume-plugins icon indicating copy to clipboard operation
docker-volume-plugins copied to clipboard
verified publisher icon marcelo-ochoa

Plugin breaks when updating from containerd 1.7.23 to 1.7.24

Open enpaul opened this issue 1 year ago • 5 comments

Describe the bug

Having this plugin installed on a system when updating from containerd-1.7.23 to containerd-1.7.24 causes docker to critically crash on service startup. On a clean install (when using containerd-1.7.24) docker cannot enable the plugin due to a critical crash. Tested with Docker 1.26 and Docker 1.27

See log output in "additional context"

To Reproduce

To reproduce "crash on install":

  1. sudo dnf install docker-ce containerd.io-1.7.24
  2. sudo systemctl start docker
  3. docker plugin install docker.io/mochoa/glusterfs-volume-plugin SERVERS=localhost --grant-all-permissions
  4. Observe error message

To reproduce "crash on upgrade":

  1. sudo dnf install docker-ce containerd.io-1.7.23
  2. sudo systemctl start docker
  3. docker plugin install docker.io/mochoa/glusterfs-volume-plugin SERVERS=localhost --grant-all-permissions
  4. sudo dnf update containerd.io
  5. sudo systemctl restart docker
  6. sudo journalctl -xe docker.service
  7. Observe failed startup in logs

Expected behavior

Server (please complete the following information):

  • OS: RockyLinux (RHEL)
  • Version: 9.5

Additional context

Failure on install error message 
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: invalid mount &{Source:/etc/ssl Destination:/etc/ssl Device:bind Flags:20480 ClearedFlags:0 PropagationFlags:[1048576] Data:readonly Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied: unknown
Failure on upgrade journalctl logs 
time="2024-12-12T00:44:34.383824108-05:00" level=info msg="Starting up"
time="2024-12-12T00:44:34.455873592-05:00" level=error msg="failed to enable plugin" error="failed to create task for container: failed to crea>
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x55d173a4a60a]
goroutine 158 [running]:
github.com/docker/docker/pkg/plugins.(*Client).callWithRetry(0x0, {0x55d174e329ab, 0x19}, {0x55d175a01d80, 0xc000725950}, 0x1, {0xc00021c588, 0>
        /root/rpmbuild/BUILD/src/engine/pkg/plugins/client.go:181 +0x14a
github.com/docker/docker/pkg/plugins.(*Client).CallWithOptions(0x0, {0x55d174e329ab, 0x19}, {0x55d1755850c0, 0x55d1770fef20}, {0x55d1753a6580, >
        /root/rpmbuild/BUILD/src/engine/pkg/plugins/client.go:134 +0x17d
github.com/docker/docker/volume/drivers.(*volumeDriverProxy).Capabilities(0xc0006e3990)
        /root/rpmbuild/BUILD/src/engine/volume/drivers/proxy.go:242 +0xe9
github.com/docker/docker/volume/drivers.(*volumeDriverAdapter).getCapabilities(0xc000725920)
        /root/rpmbuild/BUILD/src/engine/volume/drivers/adapter.go:92 +0x4b
github.com/docker/docker/volume/drivers.(*volumeDriverAdapter).Scope(0x1a?)
        /root/rpmbuild/BUILD/src/engine/volume/drivers/adapter.go:84 +0x13
github.com/docker/docker/volume/drivers.validateDriver({0x55d175a38160, 0xc000725920})
        /root/rpmbuild/BUILD/src/engine/volume/drivers/extpoint.go:118 +0x25
github.com/docker/docker/volume/drivers.(*Store).lookup(0xc000894e10, {0xc000375ec0, 0x1a}, 0x0)
        /root/rpmbuild/BUILD/src/engine/volume/drivers/extpoint.go:97 +0x2c5
github.com/docker/docker/volume/drivers.(*Store).GetDriver(...)
        /root/rpmbuild/BUILD/src/engine/volume/drivers/extpoint.go:150
github.com/docker/docker/volume/service.lookupVolume({0x55d175a2df98, 0x55d1770fef20}, 0x0?, {0xc000375ec0?, 0x0?}, {0xc000375ea0, 0x1f})
        /root/rpmbuild/BUILD/src/engine/volume/service/store.go:770 +0x73
github.com/docker/docker/volume/service.(*VolumeStore).restore.func2({{0xc000375ea0, 0x1f}, {0xc000375ec0, 0x1a}, 0xc0008bb440, 0xc0008bb470})
        /root/rpmbuild/BUILD/src/engine/volume/service/restore.go:37 +0x16d
created by github.com/docker/docker/volume/service.(*VolumeStore).restore in goroutine 1
        /root/rpmbuild/BUILD/src/engine/volume/service/restore.go:31 +0xc5
Dec 12 00:44:34 baikonur systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

enpaul avatar Dec 13 '24 00:12 enpaul

I found that this problem is caused by the bind mount options for the /etc/ssl volume, which are configured in glusterfs-volume-plugin/config.json. Changing the option "readonly" to "ro" and rebuilding the plugin, solves the issue.

BarryLuijten avatar Dec 16 '24 11:12 BarryLuijten

Performed some maintenance and a docker update broke things for me. A bit of searching led me to this. Just wanted to say this fixed it for me as well.

Zorro92 avatar Feb 13 '25 09:02 Zorro92

Thanks for the recommendation @BarryLuijten!

I'm adding my 2¢ here for anyone else with an older deployment coming across this. I have a Docker Swarm cluster with Gluster as my backing store running on Ubuntu 20.04. I needed to downgrade Barry's fork back to Gluster 7 to get everything working with that version of Ubuntu. Hope this helps someone.

CampinCarl avatar Feb 27 '25 16:02 CampinCarl

@CampinCarl are you sure it was about that Ubuntu? I just stumbled at this error while trying to update my RPi cluster and while RPi3 nodes running 64 bit are fine with the patched version built by @BarryLuijten, RPi2 nodes stuck on 32 bit are crashing like this, when trying to mount the volume:

Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=error msg="2025/06/19 01:57:05 [-s clusterpi01 -s clusterpi02 -s clusterpi03 --volfile-id=gfs --subdir-mount=/homepage/config --logger=syslog /var/lib/docker-volumes/1e65903ca234eff50299cffb
f97c5e81cb297270227116f1746429485a9ffd9e]" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[35]: [MSGID: 100030] [glusterfsd.c:2767:main] 0-glusterfs: Started
running version [{arg=glusterfs}, {version=10.1}, {cmdlinestr=glusterfs -s clusterpi01 -s clusterpi02 -s clusterpi03 --volfile-id=gfs --subdir-mount=/homepage/config --logger=syslog /var/lib/docker-volumes/1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e}]"
 plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current runn
ing process is 37" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: pending frames:" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a
1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: patchset: git://git.gluster.org/glusterfs.git" plugin=2d8b1b01
590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: signal received: 4" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f
53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: time of crash: " plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a
1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: 2025-06-19 01:57:05 +0000" plugin=2d8b1b01590ba42ce5e0a9d92f79
dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: configuration details:" plugin=2d8b1b01590ba42ce5e0a9d92f79dac
e41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: argp 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256
ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: backtrace 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d
32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: dlfcn 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d3225
6ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: libpthread 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1
d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: llistxattr 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1
d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: setfsid 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32
256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: epoll.h 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32
256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: xattr.h 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32
256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: st_atim.tv_nsec 1" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f5
3a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: package-string: glusterfs 10.1" plugin=2d8b1b01590ba42ce5e0a9d
92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="glusterfs-volume-plugin: var-lib-docker-volumes-1e65903ca234eff50299cffbf97c5e81cb297270227116f1746429485a9ffd9e[37]: ---------" plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32
256ed6ea95ccff245c29
Jun 19 03:57:05 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:05+02:00" level=info msg="Command output: " plugin=2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
Jun 19 03:57:07 clusterpi02 dockerd[24664]: time="2025-06-19T03:57:07.112546333+02:00" level=error msg="fatal task error" error="starting container failed: error while mounting volume '/var/lib/docker/plugins/2d8b1b01590ba42ce5e0a9d92f79dace41f53a1b1d32256ed6ea95ccff245c29
/rootfs': VolumeDriver.Mount: error mounting gfs/homepage/config: exit status 1" module=node/agent/taskmanager node.id=85akjv3p0quc444oj1h5tpk3y service.id=xxmdw2ddhb1y0o7mjvwd8uxws task.id=fiopuq1m2jf5wwcovajhojvnv

Both nodes are connecting to the same Gluster cluster. I just tried to build the plugin for the armv7l from this repo, just with the patched config file, but that's failing somewhere in the GO compilation🙁 In case you are also on armv7 and have the plugin compiled, it would be very kind of you if you share it😉

johny-mnemonic avatar Jun 19 '25 02:06 johny-mnemonic

For anyone with the same issue on armv7, I have rebuilt the plugin using Ubuntu 20.04 and GlusterFS 8 to make it working on this old 32bit architecture. It is available at johnymnemonic/glusterfs-volume-plugin-armv7l.

johny-mnemonic avatar Jun 24 '25 23:06 johny-mnemonic