for-linux icon indicating copy to clipboard operation
for-linux copied to clipboard

Published 20 hours ago • verified publisher icon docker

Error -> Cleanup: failed to delete container from containerd: no such container"

Open mcopjan opened this issue 4 years ago • 3 comments

I have a docker swarm on Linux machines which randomly fails with the above error, all problematic containers seems to be stuck on "Created" state.

  • [x] This is a bug report
  • [ ] This is a feature request
  • [x] I searched existing issues before opening this one

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

 Debug Mode: false
Client:

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 6
 Server Version: 19.03.5
 Storage Driver: devicemapper
  Pool Name: docker-253:2-3148406-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 10.74GB
  Backing Filesystem: xfs
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Data Space Used: 4.5GB
  Data Space Total: 107.4GB
  Data Space Available: 6.758GB
  Metadata Space Used: 25.98MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.122GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.158-RHEL7 (2019-05-13)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: wbdxb7ze6k261s33iizz4cujh
  Is Manager: false
  Node Address: 10.203.9.57
  Manager Addresses:
   10.203.9.64:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1062.9.1.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.7 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.701GiB
 Name: uiplspidoc018
 ID: 7AQI:WPM7:VVG6:ISEK:DZCP:TM4I:XVO2:2KNH:FOF2:CQMQ:C7XH:L5LI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

Output of systemctl status docker -l:

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-02-11 15:31:19 UTC; 9 months 7 days ago
     Docs: https://docs.docker.com
 Main PID: 20357 (dockerd)
    Tasks: 24
   Memory: 1.8G
   CGroup: /system.slice/docker.service
           └─20357 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.121691575Z" level=error msg="ce88f9c81bb904e4d56e6c5923d1ee5ea5ca150e1a6044efc486fbeabf530b6a cleanup: failed to delete container from containerd: no such container"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.154465891Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:9 db state:Set{{51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {84777fa48125828f0663aadcb450e4b4999abada62b81265c94f02c369851a4e 10.203.9.66 24 32 false}}"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.154521759Z" level=warning msg="Neighbor entry already present for IP 10.0.0.27, mac 02:42:0a:00:00:1b neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 27] dstMac:[2 66 10 0 0 27] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.220971243Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:9 db state:Set{{8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}}"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.266645980Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:11 db state:Set{{d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}}"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.316293487Z" level=error msg="ede21b782b5081e0af107e9d26c0db18812b7cac9f23486c87337926e6e7a805 cleanup: failed to delete container from containerd: no such container"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.413990055Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for b0685391556f33fccb0189701ea6f3b3083929b0147b0edac7fe14acd48dbedb fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key b0685391556f33fccb0189701ea6f3b3083929b0147b0edac7fe14acd48dbedb does not exist or is already being deleted"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.414024182Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome b0685391556f33fccb0189701ea6f3b3083929b0147b0edac7fe14acd48dbedb aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.463598198Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for b41e1d6402fa784ab3868ca77bf2e7a119b9daca617d6b8117071ad316271699 fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key b41e1d6402fa784ab3868ca77bf2e7a119b9daca617d6b8117071ad316271699 does not exist or is already being deleted"
Nov 18 14:52:02 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:52:02.463673390Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome b41e1d6402fa784ab3868ca77bf2e7a119b9daca617d6b8117071ad316271699 aborted lb.backEnds[eid] && lb.disabled[eid] !ok"

Output of journalctl -u docker.service | tail -n 50:

Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.598605447Z" level=error msg="ede21b782b5081e0af107e9d26c0db18812b7cac9f23486c87337926e6e7a805 cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.648608158Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:9 db state:Set{{222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {8065c2507a4c4987e6c7eb04b6565f4d01cdd09cf941f87ef0b89bda66595176 10.203.9.66 24 32 false}, {d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}}"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.648672376Z" level=warning msg="Neighbor entry already present for IP 10.0.0.27, mac 02:42:0a:00:00:1b neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 27] dstMac:[2 66 10 0 0 27] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.732126194Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:8 db state:Set{{d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}}"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.737503761Z" level=error msg="ce88f9c81bb904e4d56e6c5923d1ee5ea5ca150e1a6044efc486fbeabf530b6a cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.826695967Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 31f5485796712cf58cf4c360fb9d9790d030a9fc1d787fa84c9d35482a75576d fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key 31f5485796712cf58cf4c360fb9d9790d030a9fc1d787fa84c9d35482a75576d does not exist or is already being deleted"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.826748652Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome 31f5485796712cf58cf4c360fb9d9790d030a9fc1d787fa84c9d35482a75576d aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.886717408Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:9 db state:Set{{a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}}"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.961045350Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for e50562803b82d8f5e57d58a577b5fa7fc5bf53a36c50ec77ca04cf93321a72b5 fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key e50562803b82d8f5e57d58a577b5fa7fc5bf53a36c50ec77ca04cf93321a72b5 does not exist or is already being deleted"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.961080798Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome e50562803b82d8f5e57d58a577b5fa7fc5bf53a36c50ec77ca04cf93321a72b5 aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:55:32 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:32.961276901Z" level=error msg="ede21b782b5081e0af107e9d26c0db18812b7cac9f23486c87337926e6e7a805 cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.032708184Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:9 db state:Set{{51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {8ed070a5a3ceb8a7c21bc111998eb9e2d1a964ab1b0f0fe9b0acee84428471e6 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.032773060Z" level=warning msg="Neighbor entry already present for IP 10.0.0.27, mac 02:42:0a:00:00:1b neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 27] dstMac:[2 66 10 0 0 27] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.070622416Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:13 db state:Set{{00c045b8d68efca692a338bee16899f188c8baad67bc133c6b0481014ec5a2c0 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {023d287a3debcdf0362ba398fd0d2b83edafb95deb91e8cd48b5c9ba04286f9b 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.070734734Z" level=warning msg="Neighbor entry already present for IP 10.0.0.17, mac 02:42:0a:00:00:11 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 17] dstMac:[2 66 10 0 0 17] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.108467297Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:8 db state:Set{{d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.168498485Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for c017b0d9e89281c2990fb5eaee9aff1696ca6b9a0bd6f054774f3e757a30f93c fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key c017b0d9e89281c2990fb5eaee9aff1696ca6b9a0bd6f054774f3e757a30f93c does not exist or is already being deleted"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.168535671Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome c017b0d9e89281c2990fb5eaee9aff1696ca6b9a0bd6f054774f3e757a30f93c aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.247628848Z" level=error msg="ce88f9c81bb904e4d56e6c5923d1ee5ea5ca150e1a6044efc486fbeabf530b6a cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.269858547Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:10 db state:Set{{9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {2fb8d5111f28cb0283dbb0f0c7319a09e7f41cd5ff7c31b85dcb63461ba0e8e5 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.269930631Z" level=warning msg="Neighbor entry already present for IP 10.0.0.24, mac 02:42:0a:00:00:18 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 24] dstMac:[2 66 10 0 0 24] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.379043950Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:12 db state:Set{{20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {023d287a3debcdf0362ba398fd0d2b83edafb95deb91e8cd48b5c9ba04286f9b 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.513383155Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:11 db state:Set{{a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {2fb8d5111f28cb0283dbb0f0c7319a09e7f41cd5ff7c31b85dcb63461ba0e8e5 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {f7de13610d3e10a0363b06085fc18ad44fc9bb29c1d6ebdea60045dbf046cd0d 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.513447136Z" level=warning msg="Neighbor entry already present for IP 10.0.0.24, mac 02:42:0a:00:00:18 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 24] dstMac:[2 66 10 0 0 24] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.530295805Z" level=error msg="ede21b782b5081e0af107e9d26c0db18812b7cac9f23486c87337926e6e7a805 cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.595438565Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:9 db state:Set{{87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {d3075f2a2a5e393d48a3447996c7ad37940bc0a9d85d77c84e290a7bb1d2009a 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.595494798Z" level=warning msg="Neighbor entry already present for IP 10.0.0.27, mac 02:42:0a:00:00:1b neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 27] dstMac:[2 66 10 0 0 27] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.601116140Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for ec6185e429adbed1925f5c6c6416aa6a03dbf2f9d7fa3d27ce4480552b38c65e fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key ec6185e429adbed1925f5c6c6416aa6a03dbf2f9d7fa3d27ce4480552b38c65e does not exist or is already being deleted"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.601161736Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome ec6185e429adbed1925f5c6c6416aa6a03dbf2f9d7fa3d27ce4480552b38c65e aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.646976124Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:11 db state:Set{{34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.676824904Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:10 db state:Set{{c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {f7de13610d3e10a0363b06085fc18ad44fc9bb29c1d6ebdea60045dbf046cd0d 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.833970953Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:12 db state:Set{{0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {0e4588dec3183f2a130ad3e102d555198ce69f466ec99780f2b3edc629eba8c4 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.834032120Z" level=warning msg="Neighbor entry already present for IP 10.0.0.17, mac 02:42:0a:00:00:11 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 17] dstMac:[2 66 10 0 0 17] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.889976360Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:9 db state:Set{{91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}}"
Nov 18 14:55:33 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:33.923359872Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:8 db state:Set{{d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.020799790Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:9 db state:Set{{d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}, {87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {cfd3b61fbe36e029e4559bf33fac5133004b0748b83759505ada397eb7414517 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.020874997Z" level=warning msg="Neighbor entry already present for IP 10.0.0.27, mac 02:42:0a:00:00:1b neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 27] dstMac:[2 66 10 0 0 27] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.085218967Z" level=error msg="ce88f9c81bb904e4d56e6c5923d1ee5ea5ca150e1a6044efc486fbeabf530b6a cleanup: failed to delete container from containerd: no such container"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.123550312Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:10 db state:Set{{e89dbda8535036991baecfa000c19df5a6519e56ee65f78611d24b3200049674 10.203.9.66 24 32 false}, {8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.123610937Z" level=warning msg="Neighbor entry already present for IP 10.0.0.24, mac 02:42:0a:00:00:18 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 24] dstMac:[2 66 10 0 0 24] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.186554772Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 89e05f9425d8bd57e32e3cb10b5d58d211fd7d9abb127258942465d1c7fd0bcf fpcxplefhdm8i9vwvxbdkjz1w err:cannot delete entry endpoint_table with network id fpcxplefhdm8i9vwvxbdkjz1w and key 89e05f9425d8bd57e32e3cb10b5d58d211fd7d9abb127258942465d1c7fd0bcf does not exist or is already being deleted"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.186592552Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster test_chrome 89e05f9425d8bd57e32e3cb10b5d58d211fd7d9abb127258942465d1c7fd0bcf aborted lb.backEnds[eid] && lb.disabled[eid] !ok"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.213044199Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.27 02:42:0a:00:00:1b cardinality:8 db state:Set{{87d11c8db4031153ca414dbc3817cb486d28a4fd14f02cebd3c1cdab4dfaf525 10.203.9.66 24 32 false}, {c75bf89e61752ecb452b9da1247b209d05deb5e3aa9108f8952b6a1fe068cf8a 10.203.9.66 24 32 false}, {cfb3d5e5c594a9386703d96e3eb87d846e0f105fa296a7884a4c1d5e133da235 10.203.9.66 24 32 false}, {9bf1550c4f119e1e9db3cb7e8d2c91998729579f647e6360615154337cc20d74 10.203.9.66 24 32 false}, {51b7d692dbc1ecb555c69560ed618ea7fa391dc4aecfe8e1a317e6cf4dd3d7e4 10.203.9.66 24 32 false}, {d28c58174e76f07d555ff20510ea324ce466e08f458adb33a62bf13a0c3753fc 10.203.9.66 24 32 false}, {68a46a35005f25c4495dd19e203cc6fbfc71e64a150909ea8b660e328830df66 10.203.9.66 24 32 false}, {222196162aa571d64fa5724eb1f39f7e8a095b47e2069cb553ea0e3acc5c6551 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.249192490Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:11 db state:Set{{be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.338065915Z" level=warning msg="peerDbAdd transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:12 db state:Set{{296c6a2fe14d411be5145c8968e745874367df9a0a5d03f541784b2831347679 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}, {be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.338129727Z" level=warning msg="Neighbor entry already present for IP 10.0.0.17, mac 02:42:0a:00:00:11 neighbor:&{dstIP:[0 0 0 0 0 0 0 0 0 0 255 255 10 0 0 17] dstMac:[2 66 10 0 0 17] linkName:vx-001000-fpcxp linkDst:vxlan0 family:0} forceUpdate:false"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.426985032Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.24 02:42:0a:00:00:18 cardinality:9 db state:Set{{8ba5a6f75d3e670161c8846cbe53eb14da3ccbf083d4fc05c621e38b043e6130 10.203.9.66 24 32 false}, {91dbeb944aa752862e8a4258f1155e3f103abf6804e49a17d6c791cbb9d17cb6 10.203.9.66 24 32 false}, {9cf908734c059952dde56ce9c2817b97d8245f6d5597b06aa391791c43986549 10.203.9.66 24 32 false}, {9f926b320aa3ba40ec809ae632b24796a24df1cd04b7e74ccf30b5b746b3d1c8 10.203.9.66 24 32 false}, {a7126b279675fd672f72d8c7de4fed01cf7d51a302b798fbd38898a8692d6a76 10.203.9.66 24 32 false}, {b3712965543f3df6654c647c441fba6a68959c9c277eb61d214c216cd90a7a8a 10.203.9.66 24 32 false}, {66cceb5d800fbdaace7ed2d3008a5ceb01d638f7e2284f54a6c9e60c6e1e66ac 10.203.9.66 24 32 false}, {c64d56989612bf87d31568cad3b3641d16dba8d674388d728da5a9e0fba6a322 10.203.9.66 24 32 false}, {9f3995640eedf4a2d66c51ccbf7fdf0296a9f7c6ec1ac42b9b169628e2debafd 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.452699648Z" level=warning msg="error locating sandbox id 519fb1f5135bcfbbbbcc598fe9632712b98566dfe7c344d8d98d0b77da25e4f0: sandbox 519fb1f5135bcfbbbbcc598fe9632712b98566dfe7c344d8d98d0b77da25e4f0 not found"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.478067974Z" level=warning msg="peerDbDelete transient condition - Key:10.0.0.17 02:42:0a:00:00:11 cardinality:11 db state:Set{{be25059041c5bdd688af906605a238d861cfada2daef3089329014e73fa64da1 10.203.9.66 24 32 false}, {afca2b5070f49c95c776358785feae31da1fe975717aeee9f68358848b68b117 10.203.9.66 24 32 false}, {a6c80256b2502f525d889004ae22e28b6275110062dfdbd68ed6fd67a5d8774c 10.203.9.66 24 32 false}, {dee6903c51831a9728c72bb2557ca28d256386c0ddce021541839c9d49f33c4e 10.203.9.66 24 32 false}, {0d54056dd35900cd4a6cd26ccfdc54a4275a9e05d6d0d242600f988f6ac34e3b 10.203.9.66 24 32 false}, {0d6a248b31b7ac224a5e948b547ecaeaae9d344552daf858c764f1432d6deb57 10.203.9.66 24 32 false}, {34364464a981ce84b2f04fb1aa29d6bb8e48b1492e9e1d39b6163e9c28e62dd2 10.203.9.66 24 32 false}, {d3456cfe40c4ecbc3b97b0e71d1dff28e37818b60dc9c34bc4760f36838b5804 10.203.9.66 24 32 false}, {20c5e727e69a3011d05de2d61e37d1caf90e79d5f71684de2ce4e823af870e1b 10.203.9.66 24 32 false}, {ca8542b2e626fc203a7e1524e508f000be5a1adc7aa3f557970e6bdb8934ccc6 10.203.9.66 24 32 false}, {b050d8bbf570acadf1b3a92aa25a6db960d739c407efc9e630e056a57973ff68 10.203.9.66 24 32 false}}"
Nov 18 14:55:34 uiplspidoc018 dockerd[20357]: time="2020-11-18T14:55:34.513229128Z" level=error msg="ede21b782b5081e0af107e9d26c0db18812b7cac9f23486c87337926e6e7a805 cleanup: failed to delete container from containerd: no such container"

Workaround that helped me:

  • ssh to affected swarm node (Linux VM)
  • sudo systemctl stop docker
  • sudo systemctl start docker
  • ssh to swamr leader and run docker swarm rm {name of you swarm}
  • deploy swarm again

mcopjan avatar Nov 18 '20 15:11 mcopjan

Have recently run into this too on a clean swarm setup with one node misbehaving with:

Jun 03 05:19:09 dm1-weedplay systemd[25462]: var-lib-docker-overlay2-9291b918588a5add676e1452117315ea9e726fa34d3cbf1eb59a156d7d8e29b4-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay systemd[1]: var-lib-docker-overlay2-9291b918588a5add676e1452117315ea9e726fa34d3cbf1eb59a156d7d8e29b4-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay systemd[25462]: var-lib-docker-overlay2-af7fa67211578eabf6ef7fc9e099c65ea5c96cec0649e226fa05611f64a745a9-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay systemd[1]: var-lib-docker-overlay2-af7fa67211578eabf6ef7fc9e099c65ea5c96cec0649e226fa05611f64a745a9-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay systemd[25462]: var-lib-docker-overlay2-52471f72f4a5def02f808803db4bbeffdea547520eda78c7aca571b31395ea2a-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay dockerd[911]: time="2021-06-03T05:19:09.156432527Z" level=error msg="aff389551f311c9797ec9514dfd5af9831a46b3d0bf202346bdfc8a9acc4b90b cleanup: failed to delete container from containerd: no such container"
Jun 03 05:19:09 dm1-weedplay dockerd[911]: time="2021-06-03T05:19:09.157595028Z" level=error msg="8dab45e55974d8b32a8ac62e2e5efedaff98f3c16e4a08b9039849c725152d58 cleanup: failed to delete container from containerd: no such container"
Jun 03 05:19:09 dm1-weedplay systemd[1]: var-lib-docker-overlay2-52471f72f4a5def02f808803db4bbeffdea547520eda78c7aca571b31395ea2a-merged.mount: Succeeded.
Jun 03 05:19:09 dm1-weedplay dockerd[911]: time="2021-06-03T05:19:09.165599320Z" level=error msg="a7662ad0ab419fe24003e56468be99af87a91df9aa8dc6c97ca9390f143fd3f8 cleanup: failed to delete container from containerd: no such container"

in the docker logs, over and over.

Seems to cause quite some busy work too:

root@dm1-weedplay:~# uptime
 05:19:33 up  1:08,  1 user,  load average: 7.04, 6.46, 5.58

prologic avatar Jun 03 '21 05:06 prologic

@mcopjan Your mentioned workaround was a life saver. Thanks for taking the time to document your process. Got things working by restarting docker directly on affected nodes and redeploying swarm.

TraderSamwise avatar Jun 02 '22 23:06 TraderSamwise

I was hit by the same problem, The reboot helped, but do you know why this is happening ? Is there a root cause identified

jithinreju avatar Apr 25 '24 10:04 jithinreju