compose
compose copied to clipboard
docker
[BUG] Named volume fails to mount after reboot
Description
Hi, I have a basic EMQX stack as described in this docker-compose.yml file:
networks:
proxy:
external: true
emqx:
volumes:
emqx-data:
external: true
emqx-db:
external: true
services:
emqx1:
image: emqx/emqx:latest
container_name: emqx
restart: unless-stopped
depends_on:
- db
volumes:
- emqx-data:/opt/emqx/data
- /DOCKER/emqx/log:/opt/emqx/log
networks:
- proxy
- emqx
ports:
- ## redacted
labels:
# traefik stuff
db:
image: mysql:lts
container_name: emqx-db
restart: unless-stopped
volumes:
- emqx-db:/var/lib/mysql
- /DOCKER/emqx/config/emqx-users_schema.sql:/docker-entrypoint-initdb.d/setup.sql
networks:
- emqx
ports:
- #redacted
env_file: stack.env
I start my stack with docker compose up -d
I need the stack to restart when the host is restarted for scheduled updates.
The problem is that when the host is restarted, the docker daemon restarts all the other containers and stacks, but this one specifically, MOST TIMES does not use its named volumes. Sometimes it works correctly. New volumes are used, and since they're empty, I lose the EMQX configuration.
If I then run
docker compose down
docker compose up -d
Everything works again.
These are the volumes in question:
# docker volume inspect emqx-data
[
{
"CreatedAt": "2024-10-03T10:35:04+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/emqx-data/_data",
"Name": "emqx-data",
"Options": null,
"Scope": "local"
}
]
# docker volume inspect emqx-db
[
{
"CreatedAt": "2024-10-03T10:35:01+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/emqx-db/_data",
"Name": "emqx-db",
"Options": null,
"Scope": "local"
}
]
This is the emqx container running correctly.
# docker inspect 8920fe95927b
...
"Mounts": [
{
"Type": "bind",
"Source": "/DOCKER/emqx/log",
"Destination": "/opt/emqx/log",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "emqx-data",
"Source": "/var/lib/docker/volumes/emqx-data/_data",
"Destination": "/opt/emqx/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
...
# docker inspect 8e5d7ae3fc59
...
"Mounts": [
{
"Type": "bind",
"Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
"Destination": "/docker-entrypoint-initdb.d/setup.sql",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "emqx-db",
"Source": "/var/lib/docker/volumes/emqx-db/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
...
I rebooted 3 times. On the first 2 attempts it worked. On the third it failed. I collected this data:
# docker inspect 8920fe95927b
...
"Mounts": [
{
"Type": "volume",
"Name": "emqx-data",
"Source": "/var/lib/docker/volumes/emqx-data/_data",
"Destination": "/opt/emqx/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/DOCKER/emqx/log",
"Destination": "/opt/emqx/log",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
}
],
...
# docker inspect volume emqx-data
[
{
"CreatedAt": "2024-10-03T10:35:04+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/emqx-data/_data",
"Name": "emqx-data",
"Options": null,
"Scope": "local"
}
]
Error: No such object: volume
# docker inspect 8e5d7ae3fc59
...
"Mounts": [
{
"Type": "bind",
"Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
"Destination": "/docker-entrypoint-initdb.d/setup.sql",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "emqx-db",
"Source": "/var/lib/docker/volumes/emqx-db/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
...
# docker inspect volume emqx-db
[
{
"Id": "8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517",
"Created": "2025-05-29T08:28:51.483248553Z",
"Path": "docker-entrypoint.sh",
"Args": [
"mysqld"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1484,
"ExitCode": 0,
"Error": "",
"StartedAt": "2025-05-29T10:02:06.298292564Z",
"FinishedAt": "2025-05-29T10:01:39.319825285Z"
},
"Image": "sha256:d668aee95b72c7f64adce1bdadcf5d7ba3013105a74cd5fde1d75e0ed33e7b86",
"ResolvConfPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/hostname",
"HostsPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/hosts",
"LogPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517-json.log",
"Name": "/emqx-db",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/DOCKER/emqx/config/emqx-users_schema.sql:/docker-entrypoint-initdb.d/setup.sql:rw"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "emqx_emqx",
"PortBindings": {
"3306/tcp": [
{
"HostIp": "",
"HostPort": "13306"
}
]
},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"Mounts": [
{
"Type": "volume",
"Source": "emqx-db",
"Target": "/var/lib/mysql",
"VolumeOptions": {}
}
],
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware",
"/sys/devices/virtual/powercap"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c-init/diff:/var/lib/docker/overlay2/fe5217e7967a45e374d977a18ccd02eb639908531516cea0de5a6e4f9bb3fcdc/diff:/var/lib/docker/overlay2/d07aff2cb5dcca8dd066ac9a5cd6f83e243e9c70343503da96d1cd0b907f726d/diff:/var/lib/docker/overlay2/52d163fc3a4363e788c614f5c0342467d897a500f80aef2abc3ce2f867ff2ea2/diff:/var/lib/docker/overlay2/576cf7a6684f288ec13555fa694d40a19d6870f96813b12babd46d662a32b35c/diff:/var/lib/docker/overlay2/74be0202682fee4a13d7a7a98a62119583e11368c930ed5004d652cdf268619f/diff:/var/lib/docker/overlay2/6e9a938dc9fd3b247304b9961be9a30fd9f29265fbaeb0c0ee4432e8b0983537/diff:/var/lib/docker/overlay2/44f77f82275ca9936a6e7af866bb4bae08347c2dc710e8c19d146315dbe0f72e/diff:/var/lib/docker/overlay2/7457bfd8444df26ab2e1f2b029d66a2803cb74e8b533be3883d01c1cf59c8dc1/diff:/var/lib/docker/overlay2/ad59e608516da202fc9d0fc9300c33cde4bfd29cd2d8e98fee45451ac5040612/diff:/var/lib/docker/overlay2/cd50c55546b8d21e9215026c348e6529bf5eec066d5c72c076441784b8b1870d/diff",
"MergedDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/merged",
"UpperDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/diff",
"WorkDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
"Destination": "/docker-entrypoint-initdb.d/setup.sql",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "emqx-db",
"Source": "/var/lib/docker/volumes/emqx-db/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "8e5d7ae3fc59",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"3306/tcp": {},
"33060/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"MYSQL_ROOT_PASSWORD=ONDKaX8uBNsmDY64htsy",
"MYSQL_DATABASE=emqx-users",
"MYSQL_USER=emqx",
"MYSQL_PASSWORD=LsAx6rBjNvme2Uv7rmzV",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GOSU_VERSION=1.17",
"MYSQL_MAJOR=8.4",
"MYSQL_VERSION=8.4.2-1.el9",
"MYSQL_SHELL_VERSION=8.4.1-1.el9"
],
"Cmd": [
"mysqld"
],
"Image": "mysql:lts",
"Volumes": {
"/var/lib/mysql": {}
},
"WorkingDir": "/",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "b78c92905d2353efc20faee7289ed4d1d0ee20fe2a26e54e6db6213b01debb71",
"com.docker.compose.container-number": "1",
"com.docker.compose.depends_on": "",
"com.docker.compose.image": "sha256:d668aee95b72c7f64adce1bdadcf5d7ba3013105a74cd5fde1d75e0ed33e7b86",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "emqx",
"com.docker.compose.project.config_files": "/DOCKER/emqx/docker-compose.yml",
"com.docker.compose.project.working_dir": "/DOCKER/emqx",
"com.docker.compose.service": "db",
"com.docker.compose.version": "2.29.1"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "426475a0dff85de906cd09ea9cdb5dfe71b4102d020b592003d38c896a831d5f",
"SandboxKey": "/var/run/docker/netns/426475a0dff8",
"Ports": {
"3306/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "13306"
},
{
"HostIp": "::",
"HostPort": "13306"
}
],
"33060/tcp": null
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"emqx_emqx": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"emqx-db",
"db"
],
"MacAddress": "02:42:ac:13:00:03",
"DriverOpts": null,
"NetworkID": "e702abf9d80f387490bb880b3183d6b30342e95aa1cf8324e0ebe25980682d49",
"EndpointID": "9238ce7a442b80abf8ffe5ffe5a4ef6f611b90cf51d94bdfd668c23d9b5c08f8",
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DNSNames": [
"emqx-db",
"db",
"8e5d7ae3fc59"
]
}
}
}
}
]
Error: No such object: volume
It looks like docker is confusing the volume with the container!
Steps To Reproduce
Unsure. This is what I did:
- create named volumes
- create docker compose stack
- start with docker compose up -d
- reboot the host
- Half the times, it will fail to load the volumes.
Compose Version
Docker Compose version v2.29.1
bash: docker-compose: command not found
Docker Environment
Client: Docker Engine - Community
Version: 27.1.2
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 9
Server Version: 27.1.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-37-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.887GiB
Name: srv-docker-mqtt
ID: 5c14a396-9ba5-4241-9c82-670ffb1c2b1a
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Anything else?
I'm renaming the emqx-db container.
I think this was indeed a race condition given by the volume and the container having the same name.
Docker doesn't seem to be checking the object type.
Renaming the volume fixes the volume/container ambiguity, but does not fix the reboot problem. Upon reboot, EMQX still failed to use the saved configuration, but now the information returned by inspect is correct:
rge@srv-docker-mqtt:~$ sudo docker volume inspect emqx-db
[
{
"CreatedAt": "2024-10-03T10:35:01+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/emqx-db/_data",
"Name": "emqx-db",
"Options": null,
"Scope": "local"
}
]
Docker Compose is not involved on system reboot, it just asks engine to create containers by API, then Docker engine is in charge. According to your description, container is well set with mount "Source": "/var/lib/docker/volumes/emqx-data/_data", "Destination": "/opt/emqx/data" even when things fail, so I can't explain why container would not have access to volume content, and how a docker compose down && docker compose up (which basically stop and starts container) fixed the issue.
Can you please check if a simple docker stop emqx && docker start emqx also "fix" this issue after a volume mount failure on reboot ?
Closing due to inactivity Since we haven’t heard back in a few weeks, we’re closing this issue. Feel free to reopen or create a new one if you have more details to share.
