swarmkit icon indicating copy to clipboard operation
swarmkit copied to clipboard
verified publisher icon moby

Docker Swarm log issue caused by failure to remove an old container after docker stack deploy

Open ofirgrm opened this issue 4 years ago • 0 comments

I have a stack for Confluent Kafka (including brokers, connectors, etc..). I've restarted the stack (docker stack deploy) and run docker service logs on one of the services (cp-kafka_cp-kafka-connect1) and instead of getting the newest logs, I got the previous ones, meaning the ones that the previous container had (a container that the service task should have stopped and removed)

After the restart:

➤ docker service logs -f cp-kafka_cp-kafka-connect1 cp-kafka_cp-kafka-connect1.1.rjmam9btsc7l@cluster01 | Creating Secret File... ... cp-kafka_cp-kafka-connect1.1.rjmam9btsc7l@cluster01 | [2021-12-02 16:51:12,072] INFO records [] (uk.co.org.task.Externalprojectconnector.ExternalProjectSourceTask)

I checked where the container for this single replica service got deployed, ssh to the VM and saw that Docker Swarm only stopped the old container, but it didn't remove it:

Notice the 'exited' cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w container which should have been removed:

➤ docker ps -a |grep connect1 afc2c62ad55a confluentinc/cp-kafka-connect:latest "/connect/setup_and_…" About an hour ago Up About an hour (healthy) 8083/tcp, 9092/tcp cp-kafka_cp-kafka-connect1.1.p2wuzvcns97nywkcl6h50nfer 911d6f22a2d3 confluentinc/cp-kafka-connect:latest "/connect/setup_and_…" 4 hours ago Exited (137) About an hour ago cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

It is clear that instead of logs from cp-kafka_cp-kafka-connect1.1.p2wuzvcns97nywkcl6h50nfer I got logs from cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

I then remove the 'exited' container: ➤ docker rm cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

And again run: docker service logs cp-kafka_cp-kafka-connect1 This time it was showing the latest logs as expected:

2021-12-02T16:52:04.443428257Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | Creating Secret File... 2021-12-02T16:52:04.446932866Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | Starting Kafka Connect... ... 2021-12-02T18:24:32.230561770Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | [2021-12-02 18:24:32,230] INFO records [] (uk.co.org.task.Externalprojectconnector.ExternalProjectSourceTask)

➤ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION mllkj8tx019ml4jhuelduhmju * cluster01 Ready Active Reachable 19.03.7 p6vdnwc7ahaogs09gfif70m2l cluster02 Ready Active Leader 19.03.7 vstjjddqusu2nw5rztt967mtq cluster03 Ready Active Reachable 19.03.7

➤ docker version Client: Docker Engine - Community Version: 19.03.6 API version: 1.40 Go version: go1.12.16 Git commit: 369ce74a3c Built: Thu Feb 13 01:27:49 2020 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.7 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: 7141c199a2 Built: Wed Mar 4 01:21:08 2020 OS/Arch: linux/amd64 Experimental: true containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683

➤ docker info Client: Debug Mode: false

Server: Containers: 19 Running: 19 Paused: 0 Stopped: 0 Images: 23 Server Version: 19.03.7 Storage Driver: overlay2 Backing Filesystem: Supports d_type: true Native Overlay Diff: true Logging Driver: local Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: active NodeID: mllkj8tx019ml4jhuelduhmju Is Manager: true ClusterID: pxrog2yr4vbj8tc0ndhf3j680 Managers: 3 Nodes: 5 Default Address Pool: 10.0.0.0/8 SubnetSize: 24 Data Path Port: 4789 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 10.12.1.215 Manager Addresses: 10.12.1.215:2377 10.12.1.216:2377 10.12.1.222:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-147-generic Operating System: Ubuntu 18.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 23.54GiB Name: cluster01 ID: JBN4:SSQD:K72A:USUE:3UPI:OG72:3IXQ:E7JJ:Q2RG:QATP:2RCQ:FEUW Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: nexus:9082 nexus:9083 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support

ofirgrm avatar Dec 02 '21 18:12 ofirgrm