for-linux icon indicating copy to clipboard operation
for-linux copied to clipboard

Published 20 hours ago • verified publisher icon docker

memberlist: Was able to connect to b3e0c29eb2bf but other probes failed, network may be misconfigured

Open majiajue opened this issue 5 years ago • 3 comments

  • [x] This is a bug report
  • [ ] This is a feature request
  • [ ] I searched existing issues before opening this one

Expected behavior

Actual behavior

Steps to reproduce the behavior

Output of docker version:

OS:centos 7.7
kernel: Linux 3.10.0-1062.1.2.el7.x86_64
Docker version 19.03.3,

Output of docker info:


Client:
 Debug Mode: false

Server:
 Containers: 113
  Running: 11
  Paused: 0
  Stopped: 102
 Images: 181
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 5x22o7ol1eaa9x4ldatd9mo47
  Is Manager: false
  Node Address: 172.40.1.31
  Manager Addresses:
   172.40.1.29:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1062.1.2.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.74GiB
 Name: docker03.crm.konka.com
 ID: SQ54:77BK:QJCX:7AIV:CQO2:7OP4:JYWB:PWBN:6PBS:N5MN:555W:7GBK
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  192.168.9.139
  172.40.1.71
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
You have mail in /var/spool/mail/root

Additional environment details (AWS, VirtualBox, physical, etc.)

Dec 18 14:04:07 docker03 dockerd: time="2019-12-18T14:04:07.456353508+08:00" level=error msg="Bulk sync to node 846ab1c1006d timed out"
Dec 18 14:04:07 docker03 dockerd: time="2019-12-18T14:04:07.461087826+08:00" level=warning msg="memberlist: Push/Pull with 846ab1c1006d failed: dial tcp 172.40.1.30:7946: i/o timeout"
Dec 18 14:04:08 docker03 dockerd: time="2019-12-18T14:04:08.158591113+08:00" level=info msg="memberlist: Suspect b3e0c29eb2bf has failed, no acks received"
Dec 18 14:04:08 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:09 docker03 dockerd: time="2019-12-18T14:04:09.880111879+08:00" level=warning msg="memberlist: Failed to push local state: write tcp 172.40.1.31:7946->172.40.1.30:34636: i/o timeout from=172.40.1.30:34636"
Dec 18 14:04:09 docker03 dockerd: time="2019-12-18T14:04:09.887325001+08:00" level=warning msg="memberlist: Failed to push local state: write tcp 172.40.1.31:7946->172.40.1.30:34592: i/o timeout from=172.40.1.30:34592"
Dec 18 14:04:10 docker03 dockerd: time="2019-12-18T14:04:10.079633931+08:00" level=warning msg="memberlist: Refuting a suspect message (from: e96a082330cd)"
Dec 18 14:04:10 docker03 dockerd: time="2019-12-18T14:04:10.121728549+08:00" level=info msg="Node b3e0c29eb2bf/172.40.1.29, joined gossip cluster"
Dec 18 14:04:10 docker03 dockerd: time="2019-12-18T14:04:10.585831028+08:00" level=info msg="Node b3e0c29eb2bf change state NodeFailed --> NodeActive"
Dec 18 14:04:11 docker03 metricbeat: 2019-12-18T14:04:11.948+0800#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":79221720,"time":{"ms":1713}},"total":{"ticks":174161230,"time":{"ms":1907},"value":174161230},"user":{"ticks":94939510,"time":{"ms":194}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":17},"info":{"ephemeral_id":"261099d3-8524-40d6-b5e5-71afd6d36423","uptime":{"ms":4913190076}},"memstats":{"gc_next":60566288,"memory_alloc":45055288,"memory_total":23065083398704,"rss":5210112},"runtime":{"goroutines":72}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":109,"batches":7,"total":109},"read":{"bytes":3076},"write":{"bytes":214018}},"pipeline":{"clients":4,"events":{"active":0,"published":104,"total":104},"queue":{"acked":109}}},"metricbeat":{"docker":{"cpu":{"events":2,"failures":2},"diskio":{"events":2,"failures":2},"memory":{"events":2,"failures":2},"network":{"events":2,"failures":2}},"system":{"cpu":{"events":2,"success":2},"load":{"events":2,"success":2},"memory":{"events":2,"success":2},"network":{"events":48,"success":48},"process":{"events":34,"success":34},"process_summary":{"events":4,"success":4},"socket_summary":{"events":4,"success":4}}},"system":{"load":{"1":96.26,"15":14.06,"5":37.72,"norm":{"1":6.0163,"15":0.8788,"5":2.3575}}}}}}
Dec 18 14:04:13 docker03 dockerd: time="2019-12-18T14:04:13.258070834+08:00" level=warning msg="memberlist: Was able to connect to 846ab1c1006d but other probes failed, network may be misconfigured"
Dec 18 14:04:14 docker03 dockerd: time="2019-12-18T14:04:14.689017916+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:14 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:14 docker03 dockerd: time="2019-12-18T14:04:14.751054259+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:14 docker03 dockerd: time="2019-12-18T14:04:14.894519242+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:14 docker03 dockerd: time="2019-12-18T14:04:14.790618485+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:15 docker03 dockerd: time="2019-12-18T14:04:14.999436116+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:15 docker03 dockerd: time="2019-12-18T14:04:15.020111657+08:00" level=error msg="agent: session failed" backoff=100ms error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=node/agent node.id=5x22o7ol1eaa9x4ldatd9mo47
Dec 18 14:04:15 docker03 dockerd: time="2019-12-18T14:04:15.027319774+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:15 docker03 metricbeat: 2019-12-18T14:04:15.084+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.memory: failed to get docker stats: context deadline exceeded
Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:15 docker03 metricbeat: 2019-12-18T14:04:15.231+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.cpu: failed to get docker stats: context deadline exceeded
Dec 18 14:04:15 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:15 docker03 metricbeat: 2019-12-18T14:04:15.231+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.diskio: failed to get docker stats: context deadline exceeded
Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)

majiajue avatar Dec 23 '19 03:12 majiajue 

Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:15 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:15 docker03 metricbeat: 2019-12-18T14:04:15.424+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.network: failed to get docker stats: context deadline exceeded
Dec 18 14:04:16 docker03 dockerd: time="2019-12-18T14:04:16.020707060+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:16 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:16 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.153502928+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:17 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.562077848+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.779206695+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.810371326+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.842716704+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:17 docker03 dockerd: time="2019-12-18T14:04:17.981131594+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:17 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.001852436+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.083318193+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.083674635+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.092067177+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.092067273+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.138953008+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.138965703+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.198166016+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.229853851+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.210562152+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.258257967+08:00" level=warning msg="memberlist: Was able to connect to 846ab1c1006d but other probes failed, network may be misconfigured"
Dec 18 14:04:18 docker03 dockerd: time="2019-12-18T14:04:18.312227792+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:18 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:19 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:19 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:20 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:35 docker03 metricbeat: 2019-12-18T14:04:35.117+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.memory: failed to get docker stats: context deadline exceeded
Dec 18 14:04:35 docker03 metricbeat: 2019-12-18T14:04:35.293+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.cpu: failed to get docker stats: context deadline exceeded
Dec 18 14:04:35 docker03 metricbeat: 2019-12-18T14:04:35.293+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.diskio: failed to get docker stats: context deadline exceeded
Dec 18 14:04:35 docker03 metricbeat: 2019-12-18T14:04:35.425+0800#011INFO#011module/wrapper.go:252#011Error fetching data for metricset docker.network: failed to get docker stats: context deadline exceeded
Dec 18 14:04:39 docker03 dockerd: time="2019-12-18T14:04:39.172712214+08:00" level=warning msg="memberlist: Was able to connect to b3e0c29eb2bf but other probes failed, network may be misconfigured"
Dec 18 14:04:39 docker03 dockerd: time="2019-12-18T14:04:39.653561061+08:00" level=warning msg="xtables contention detected while running [--wait -t filter -nL DOCKER-INGRESS]: Waited for 19.06 seconds and received \"Chain DOCKER-INGRESS (1 references)\\ntarget     prot opt source               destination         \\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:19000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:19000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:17000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:7000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8761\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8761\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:9000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8090\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8090\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8763\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8763\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:39092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:39092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:29092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:29092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:10000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:10000\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9093\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:9093\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:19092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:19092\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:22181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9161\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:9161\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17004\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:17004\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7004\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:7004\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:12181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:12181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:32181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:32181\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8080\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8080\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17001\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:17001\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7001\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:7001\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:443\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:80\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17005\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:17005\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7005\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:7005\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17002\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:17002\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7002\\nACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:7002\\nRETURN     all  --  0.0.0.0/0            0.0.0.0/0           \\n\""
Dec 18 14:04:39 docker03 dockerd: time="2019-12-18T14:04:39.800722905+08:00" level=warning msg="NetworkDB stats docker03.crm.konka.com(e96a082330cd) - healthscore:3 (connectivity issues)"
Dec 18 14:04:39 docker03 dockerd: time="2019-12-18T14:04:39.686880494+08:00" level=warning msg="memberlist: Refuting a suspect message (from: e96a082330cd)"
Dec 18 14:04:40 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:40 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:40 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:41 docker03 dockerd: time="2019-12-18T14:04:41.062876705+08:00" level=error msg="Bulk sync to node 846ab1c1006d timed out"
Dec 18 14:04:41 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:41 docker03 dockerd: time="2019-12-18T14:04:41.882531570+08:00" level=info msg="parsed scheme: \"\"" module=grpc
Dec 18 14:04:41 docker03 dockerd: time="2019-12-18T14:04:41.882587399+08:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Dec 18 14:04:41 docker03 metricbeat: 2019-12-18T14:04:41.936+0800#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":79222260,"time":{"ms":542}},"total":{"ticks":174162730,"time":{"ms":1494},"value":174162730},"user":{"ticks":94940470,"time":{"ms":952}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":16},"info":{"ephemeral_id":"261099d3-8524-40d6-b5e5-71afd6d36423","uptime":{"ms":4913220063}},"memstats":{"gc_next":57725072,"memory_alloc":43656200,"memory_total":23065250439264,"rss":1667072},"runtime":{"goroutines":72}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":208,"batches":13,"total":208},"read":{"bytes":5763},"write":{"bytes":347414}},"pipeline":{"clients":4,"events":{"active":0,"published":208,"total":208},"queue":{"acked":208}}},"metricbeat":{"docker":{"container":{"events":51,"success":51},"cpu":{"events":19,"failures":2,"success":17},"diskio":{"events":19,"failures":2,"success":17},"info":{"events":4,"success":4},"memory":{"events":2,"failures":2},"network":{"events":2,"failures":2}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":72,"success":72},"process":{"events":24,"success":24},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":70.95,"15":14.63,"5":37.17,"norm":{"1":4.4344,"15":0.9144,"5":2.3231}}}}}}
Dec 18 14:04:42 docker03 dockerd: time="2019-12-18T14:04:42.203934438+08:00" level=warning msg="memberlist: Was able to connect to b3e0c29eb2bf but other probes failed, network may be misconfigured"
Dec 18 14:04:42 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:42 docker03 dockerd: time="2019-12-18T14:04:42.312695694+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:42 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:42 docker03 dockerd: time="2019-12-18T14:04:42.497890503+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:42 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:42 docker03 dockerd: time="2019-12-18T14:04:42.996531195+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:43 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:43 docker03 dockerd: time="2019-12-18T14:04:43.495899924+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{172.40.1.29:2377 0  <nil>}] <nil>}" module=grpc
Dec 18 14:04:43 docker03 dockerd: time="2019-12-18T14:04:43.495944799+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 18 14:04:43 docker03 dockerd: time="2019-12-18T14:04:43.674937958+08:00" level=info msg="manager selected by agent for new session: {chr59ufheiq7ryc9992rk7v1m 172.40.1.29:2377}" module=node/agent node.id=5x22o7ol1eaa9x4ldatd9mo47
Dec 18 14:04:43 docker03 dockerd: time="2019-12-18T14:04:43.675927289+08:00" level=info msg="waiting 63.744649ms before registering session" module=node/agent node.id=5x22o7ol1eaa9x4ldatd9mo47
Dec 18 14:04:43 docker03 dockerd: time="2019-12-18T14:04:43.980995203+08:00" level=error msg="Bulk sync to node b3e0c29eb2bf timed out"
Dec 18 14:04:44 docker03 dockerd: time="2019-12-18T14:04:44.205499667+08:00" level=error msg="Handler for GET /v1.31/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Dec 18 14:04:44 docker03 dockerd: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
Dec 18 14:04:44 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:44 docker03 kernel: IPVS: __ip_vs_del_service: enter
Dec 18 14:04:45 docker03 kernel: IPVS: __ip_vs_del_service: enter

majiajue avatar Dec 23 '19 03:12 majiajue

Hello, I've come enough of this problem. Has the problem been solved?

NadamHL avatar Dec 21 '20 03:12 NadamHL

@NadamHL and @majiajue did this issue persist with latest versions of Docker ? ( > 24.0 )

Also could you please share your kernel configurations ? Thanks :)

redotics avatar Nov 23 '23 15:11 redotics