i/o timeout between controller and server

[shcorya]

I'm running in docker swarm with an overlay network connecting the containers as in the example.

The server listens on the IP address.

caddy_proxy.0.lzmv38dhex0h@us-las-021-01    | {"level":"info","ts":1726160108.0723393,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_proxy.0.lzmv38dhex0h@us-las-021-01    | {"level":"info","ts":1726160108.0732574,"logger":"admin","msg":"admin endpoint started","address":"10.201.200.3:2019","enforce_origin":false,"origins":["//10.201.200.3:2019"]}
caddy_proxy.0.lzmv38dhex0h@us-las-021-01    | {"level":"info","ts":1726160108.0735753,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_proxy.0.gyzzb6bsg2tl@us-nyc-021-01    | {"level":"info","ts":1726160138.1796622,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_proxy.0.gyzzb6bsg2tl@us-nyc-021-01    | {"level":"info","ts":1726160138.181562,"logger":"admin","msg":"admin endpoint started","address":"10.201.200.5:2019","enforce_origin":false,"origins":["//10.201.200.5:2019"]}
caddy_proxy.0.gyzzb6bsg2tl@us-nyc-021-01    | {"level":"info","ts":1726160138.1820304,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_proxy.0.047azmgvq20i@us-mia-021-01    | {"level":"info","ts":1726160106.8104422,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_proxy.0.047azmgvq20i@us-mia-021-01    | {"level":"info","ts":1726160106.8117878,"logger":"admin","msg":"admin endpoint started","address":"10.201.200.4:2019","enforce_origin":false,"origins":["//10.201.200.4:2019"]}
caddy_proxy.0.047azmgvq20i@us-mia-021-01    | {"level":"info","ts":1726160106.812302,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_proxy.0.hgfp9hnxdam8@lu-bsn-155-01    | {"level":"info","ts":1726160071.3319218,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_proxy.0.hgfp9hnxdam8@lu-bsn-155-01    | {"level":"info","ts":1726160071.3332376,"logger":"admin","msg":"admin endpoint started","address":"10.201.200.6:2019","enforce_origin":false,"origins":["//10.201.200.6:2019"]}
caddy_proxy.0.hgfp9hnxdam8@lu-bsn-155-01    | {"level":"info","ts":1726160071.3337276,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}

But the controller times out when sending the configuration.

caddy_controller.1.1h95urn5i5n7@nl-ams-155-03    | {"level":"error","ts":1726162052.0135264,"logger":"docker-proxy","msg":"Failed to send configuration to","server":"10.201.200.3","error":"Post \"http://10.201.200.3:2019/load\": dial tcp 10.201.200.3:2019: i/o timeout"}
caddy_controller.1.1h95urn5i5n7@nl-ams-155-03    | {"level":"error","ts":1726162052.0148907,"logger":"docker-proxy","msg":"Failed to send configuration to","server":"10.201.200.4","error":"Post \"http://10.201.200.4:2019/load\": dial tcp 10.201.200.4:2019: i/o timeout"}
caddy_controller.1.1h95urn5i5n7@nl-ams-155-03    | {"level":"error","ts":1726162052.0150003,"logger":"docker-proxy","msg":"Failed to send configuration to","server":"10.201.200.5","error":"Post \"http://10.201.200.5:2019/load\": dial tcp 10.201.200.5:2019: i/o timeout"}
caddy_controller.1.1h95urn5i5n7@nl-ams-155-03    | {"level":"error","ts":1726162052.0161746,"logger":"docker-proxy","msg":"Failed to send configuration to","server":"10.201.200.6","error":"Post \"http://10.201.200.6:2019/load\": dial tcp 10.201.200.6:2019: i/o timeout"}

[shcorya]

I fixed it. I had an extraneous overlay network connected to the swarm services. Removing this network fixed the issue.