for-linux
for-linux copied to clipboard
docker
docker service update --force SERVICE changes mounts order
- [x] This is a bug report
- [ ] This is a feature request
- [ ] I searched existing issues before opening this one
Expected behavior
A forced update of a service should not change the mount order.
Actual behavior
When running a forced update of a service to restart it the order of the mounts change, this causes unexpected service restarts when re-deploying the stack.
Steps to reproduce the behavior
Using this compose-file:
$ cat mount-issue.yml
version: '3.5'
services:
hello-world:
image: woahbase/alpine-supervisor
volumes:
- type: tmpfs
target: /tmp
- /var/log:/test/log
- /var/log:/test/data/import
- /var/log:/test/data/blobstore
- /var/log:/test/data/license
- /var/log:/test/auth
- /var/log:/test/release
- /var/log:/test/data/test-ws/exports
- Stack deploy:
$ docker stack deploy -c mount-issue.yml mount
- Forced service update:
$ docker service update --force mount_hello-world
- Another stack deploy:
$ docker stack deploy -c mount-issue.yml mount
One can now see that three tasks has been created, although the last deploy should not have resulted in a new task as nothing changed from the first deploy.
$ docker service ps -q mount_hello-world
4g1tu5w3bndusu4nl4awuyiqe
q9xvpg7paks4fde44o0ejc6iv
752ntfjb3bor9xfua0ozxkete
When comparing the output of docker inspect for the first and the second task one can see that the mounts have changed order:
$ diff docker-inspect-first-task docker-inspect-second-task
3c3
< "ID": "752ntfjb3bor9xfua0ozxkete",
---
> "ID": "q9xvpg7paks4fde44o0ejc6iv",
5c5
< "Index": 392134
---
> "Index": 392145
7,8c7,8
< "CreatedAt": "2020-01-09T09:22:39.776079322Z",
< "UpdatedAt": "2020-01-09T09:25:05.412734209Z",
---
> "CreatedAt": "2020-01-09T09:25:00.993867387Z",
> "UpdatedAt": "2020-01-09T09:26:04.369133635Z",
28c28
< "Target": "/test/log"
---
> "Target": "/test/auth"
33c33
< "Target": "/test/data/import"
---
> "Target": "/test/data/blobstore"
38c38
< "Target": "/test/data/blobstore"
---
> "Target": "/test/data/import"
48c48
< "Target": "/test/auth"
---
> "Target": "/test/data/test-ws/exports"
53c53
< "Target": "/test/release"
---
> "Target": "/test/log"
58c58
< "Target": "/test/data/test-ws/exports"
---
> "Target": "/test/release"
80c80
< "ForceUpdate": 0
---
> "ForceUpdate": 1
86c86
< "Timestamp": "2020-01-09T09:25:05.322331467Z",
---
> "Timestamp": "2020-01-09T09:26:04.344980584Z",
90c90
< "ContainerID": "9b2797574a201e962544545e7abf3f617fa023a94832246b653ab921c0279884",
---
> "ContainerID": "3e9f2e74a428a058fee8d8593e39cfc9f2f982dd8f13f5e2771e862f40e29813",
135c135
< "10.0.19.3/24"
---
> "10.0.19.5/24"
However, when comparing the output of docker inspect for the first and the third task one can see that the mounts are back in order:
$ diff docker-inspect-first-task docker-inspect-third-task
3c3
< "ID": "752ntfjb3bor9xfua0ozxkete",
---
> "ID": "4g1tu5w3bndusu4nl4awuyiqe",
5c5
< "Index": 392134
---
> "Index": 392148
7,8c7,8
< "CreatedAt": "2020-01-09T09:22:39.776079322Z",
< "UpdatedAt": "2020-01-09T09:25:05.412734209Z",
---
> "CreatedAt": "2020-01-09T09:26:00.104479854Z",
> "UpdatedAt": "2020-01-09T09:26:05.875837575Z",
80c80
< "ForceUpdate": 0
---
> "ForceUpdate": 1
86,88c86,88
< "Timestamp": "2020-01-09T09:25:05.322331467Z",
< "State": "shutdown",
< "Message": "shutdown",
---
> "Timestamp": "2020-01-09T09:26:05.817989009Z",
> "State": "running",
> "Message": "started",
90,91c90,91
< "ContainerID": "9b2797574a201e962544545e7abf3f617fa023a94832246b653ab921c0279884",
< "PID": 0,
---
> "ContainerID": "35e681cc36a39c57ab62ded1b33c75021c886ba4eb9dabf4c1d6e995eaf07ff2",
> "PID": 21090,
96c96
< "DesiredState": "shutdown",
---
> "DesiredState": "running",
135c135
< "10.0.19.3/24"
---
> "10.0.19.6/24"
Output of docker version:
$ docker version
Client: Docker Engine - Community
Version: 19.03.3
API version: 1.40
Go version: go1.12.10
Git commit: a872fc2f86
Built: Tue Oct 8 01:00:44 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.3
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: a872fc2f86
Built: Tue Oct 8 00:59:17 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
$ docker info
Client:
Debug Mode: false
Server:
Containers: 26
Running: 24
Paused: 0
Stopped: 2
Images: 343
Server Version: 19.03.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: 1mq1c0kp9qsfznncxsi5x9adb
Is Manager: true
ClusterID: buvaarnh3paslybdpg3qmuavj
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.101.199
Manager Addresses:
192.168.101.199:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-37-generic
Operating System: Ubuntu 19.04
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.08GiB
Name: viktor-Precision-5530
ID: 4Y75:LKUS:LXZN:23QS:QIZK:SF5L:ZZK3:2KUC:5EOS:YNJQ:N5OA:YKW3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
local.seal-software.net:5443
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.) Physical machine
Observed with non-forced update as well, which seems more problematic.
docker version
Client: Docker Engine - Community
Version: 26.1.1
API version: 1.45
Go version: go1.21.9
Git commit: 4cf5afa
Built: Tue Apr 30 11:47:53 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.1
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: ac2de55
Built: Tue Apr 30 11:47:53 2024
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
@viktor-thell-seal , @camhorn , did anyone of ever find a solution/workaround for the problem? It seems that it still exists in recent (version 28.x) docker versions.
