telepresence Unable to reach services in cluster

Unable to reach services in cluster

Open zquintana opened this issue 2 years ago • 3 comments

Describe the bug I'm unable to get reach any services through the proxy. Started telepresence with connect an intercept, I also added the config for also-proxy like so:

- cluster:
    certificate-authority-data: ~ # truncated
    extensions:
    - extension:
        also-proxy:
        - 172.20.0.0/16
        - 10.150.232.52/32
        manager:
          namespace: ambassador
      name: telepresence.io
    server: ~ # truncated
  name: ~ # truncated

DNS is resolving properly to IP addresses in the cluster. Example nslookup service_name yield the expected IP addresses. Still actually trying to curl them I get:

*   Trying 172.20.91.195:8000...
* TCP_NODELAY set
* connect to 172.20.91.195 port 8000 failed: No route to host
* Failed to connect to my_service.my_ns port 8000: No route to host
* Closing connection 0
curl: (7) Failed to connect to my_service.my_ns port 8000: No route to host

To Reproduce Steps to reproduce the behavior:

telepresence connect
curl my_service.my_namespace:8000
See error above

Expected behavior I expect that the curl should respond with a response from the service.

Versions (please complete the following information): Telepresence Client: v2.5.8 (api v3) Root Daemon: v2.5.8 (api v3) User Daemon: v2.5.8 (api v3)

Kubernetes

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

May 10 '22 19:05 zquintana

I have similar issue when using AWS EKS. nslookup successfully resolves IP, but curl fails

Ip address of web services is 172.20.98.4

~$: kubectl get svc -o wide | grep web 
web                      ClusterIP   172.20.98.4      <none>        80/TCP         28d     app=web
web-bff                  NodePort    172.20.185.56    <none>        80:30213/TCP   29d     app=web-bff

It has been resolved correctly

~$: nslookup web.default
Server:		192.168.64.1
Address:	192.168.64.1#53

Name:	web.default
Address: 172.20.98.4

But curl doesn't connect

~$: curl http://web.default/
curl: (52) Empty reply from server
~$: curl web.default:80
curl: (52) Empty reply from server
~$:

The same issue when I try to connect to Redis service.

~ $ redis-cli -h redis.default
redis.default:6379> ECHO 123
Error: Server closed the connection
(3.38s)
redis.default:6379> keys *
Error: Server closed the connection
(3.23s)
redis.default:6379>

daemon.log

~ $ tail -f ~/.cache/telepresence/logs/daemon.log 
2022-05-12 09:56:18.7428 info    daemon/session/dns/legacy/Server:shutdown_logger : shutting down (gracefully)...
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6869 msg="started command [\"iptables\" \"-t\" \"nat\" \"-D\" \"OUTPUT\" \"-j\" \"TELEPRESENCE_DNS\"]"
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6869 dexec.stream=stdin dexec.err=EOF
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6869 msg="finished successfully: exit status 0"
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6870 msg="started command [\"iptables\" \"-t\" \"nat\" \"-F\" \"TELEPRESENCE_DNS\"]"
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6870 dexec.stream=stdin dexec.err=EOF
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6870 msg="finished successfully: exit status 0"
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6871 msg="started command [\"iptables\" \"-t\" \"nat\" \"-X\" \"TELEPRESENCE_DNS\"]"
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6871 dexec.stream=stdin dexec.err=EOF
time="2022-05-12T09:56:18+02:00" level=info dexec.pid=6871 msg="finished successfully: exit status 0"


2022-05-12 09:56:53.1872 info    daemon/session : Connected to Manager 2.5.8
2022-05-12 09:56:53.2092 info    daemon/session : Adding never-proxy subnet 52.43.210.66/32
2022-05-12 09:56:53.2108 info    daemon/session : Adding never-proxy subnet 52.39.131.228/32
2022-05-12 09:56:55.4046 info    daemon/session/watch-cluster-info : Setting cluster DNS to 172.20.0.10
2022-05-12 09:56:55.4047 info    daemon/session/watch-cluster-info : Setting cluster domain to "cluster.local."
2022-05-12 09:56:55.4047 info    daemon/session/watch-cluster-info : Adding service subnet 172.20.0.0/16
2022-05-12 09:56:55.4047 info    daemon/session/watch-cluster-info : Adding pod subnet 10.0.0.0/22
2022-05-12 09:56:55.4052 info    daemon/session/watch-cluster-info : started command ["ip" "a" "add" "172.20.0.0/16" "dev" "tel0"] : dexec.pid="7500"
2022-05-12 09:56:55.4053 info    daemon/session/watch-cluster-info :  : dexec.pid="7500" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4066 error   daemon/session/dns/resolved : systemd-resolved is not running
2022-05-12 09:56:55.4066 info    daemon/session/dns : Unable to use systemd-resolved, falling back to local server
2022-05-12 09:56:55.4066 info    daemon/session/watch-cluster-info : finished successfully: exit status 0 : dexec.pid="7500"
2022-05-12 09:56:55.4067 info    daemon/session/dns/legacy : Automatically set -dns=192.168.64.1
2022-05-12 09:56:55.4070 info    daemon/session/watch-cluster-info : started command ["ip" "a" "add" "10.0.0.0/22" "dev" "tel0"] : dexec.pid="7501"
2022-05-12 09:56:55.4071 info    daemon/session/watch-cluster-info :  : dexec.pid="7501" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4074 info    daemon/session/dns/legacy : started command ["docker" "inspect" "bridge" "-f" "{{(index .IPAM.Config 0).Gateway}}"] : dexec.pid="7502"
2022-05-12 09:56:55.4075 info    daemon/session/dns/legacy :  : dexec.pid="7502" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4085 info    daemon/session/watch-cluster-info : finished successfully: exit status 0 : dexec.pid="7501"
2022-05-12 09:56:55.4354 info    daemon/session/dns/legacy :  : dexec.pid="7502" dexec.stream="stdout" dexec.data="172.17.0.1\n"
2022-05-12 09:56:55.4377 info    daemon/session/dns/legacy : finished successfully: exit status 0 : dexec.pid="7502"
2022-05-12 09:56:55.4379 info    daemon/session/dns/legacy : listening to docker bridge at 172.17.0.1
2022-05-12 09:56:55.4397 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-D" "OUTPUT" "-j" "TELEPRESENCE_DNS"] : dexec.pid="7521"
2022-05-12 09:56:55.4398 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7521" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4404 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7521" dexec.stream="stdout+stderr" dexec.data="iptables v1.8.7 (nf_tables): Chain 'TELEPRESENCE_DNS' does not exist\n"
2022-05-12 09:56:55.4405 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7521" dexec.stream="stdout+stderr" dexec.data="Try `iptables -h' or 'iptables --help' for more information.\n"
2022-05-12 09:56:55.4405 info    daemon/session/dns/legacy/NAT-redirect : finished with error: exit status 2 : dexec.pid="7521"
2022-05-12 09:56:55.4407 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-F" "TELEPRESENCE_DNS"] : dexec.pid="7523"
2022-05-12 09:56:55.4408 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7523" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4411 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7523" dexec.stream="stdout+stderr" dexec.data="iptables: No chain/target/match by that name.\n"
2022-05-12 09:56:55.4412 info    daemon/session/dns/legacy/NAT-redirect : finished with error: exit status 1 : dexec.pid="7523"
2022-05-12 09:56:55.4416 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-X" "TELEPRESENCE_DNS"] : dexec.pid="7524"
2022-05-12 09:56:55.4416 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7524" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4423 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7524" dexec.stream="stdout+stderr" dexec.data="iptables: No chain/target/match by that name.\n"
2022-05-12 09:56:55.4425 info    daemon/session/dns/legacy/NAT-redirect : finished with error: exit status 1 : dexec.pid="7524"
2022-05-12 09:56:55.4427 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-N" "TELEPRESENCE_DNS"] : dexec.pid="7525"
2022-05-12 09:56:55.4428 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7525" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4436 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7525"
2022-05-12 09:56:55.4439 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "43899" "-j" "RETURN"] : dexec.pid="7526"
2022-05-12 09:56:55.4439 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7526" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4453 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7526"
2022-05-12 09:56:55.4456 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "60956" "-j" "RETURN"] : dexec.pid="7527"
2022-05-12 09:56:55.4457 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7527" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4464 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7527"
2022-05-12 09:56:55.4467 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "41821" "-j" "RETURN"] : dexec.pid="7528"
2022-05-12 09:56:55.4468 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7528" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4482 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7528"
2022-05-12 09:56:55.4485 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "35218" "-j" "RETURN"] : dexec.pid="7529"
2022-05-12 09:56:55.4486 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7529" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4498 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7529"
2022-05-12 09:56:55.4501 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "35867" "-j" "RETURN"] : dexec.pid="7530"
2022-05-12 09:56:55.4501 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7530" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4513 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7530"
2022-05-12 09:56:55.4515 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "48946" "-j" "RETURN"] : dexec.pid="7531"
2022-05-12 09:56:55.4516 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7531" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4522 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7531"
2022-05-12 09:56:55.4524 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "46664" "-j" "RETURN"] : dexec.pid="7532"
2022-05-12 09:56:55.4525 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7532" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4531 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7532"
2022-05-12 09:56:55.4533 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "54752" "-j" "RETURN"] : dexec.pid="7533"
2022-05-12 09:56:55.4534 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7533" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4545 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7533"
2022-05-12 09:56:55.4547 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "48524" "-j" "RETURN"] : dexec.pid="7534"
2022-05-12 09:56:55.4548 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7534" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4554 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7534"
2022-05-12 09:56:55.4556 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--source" "192.168.64.106" "--sport" "44420" "-j" "RETURN"] : dexec.pid="7535"
2022-05-12 09:56:55.4557 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7535" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4568 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7535"
2022-05-12 09:56:55.4571 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-A" "TELEPRESENCE_DNS" "-p" "udp" "--dest" "192.168.64.1/32" "--dport" "53" "-j" "DNAT" "--to-destination" "127.0.0.1:34221"] : dexec.pid="7536"
2022-05-12 09:56:55.4572 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7536" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4585 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7536"
2022-05-12 09:56:55.4588 info    daemon/session/dns/legacy/NAT-redirect : started command ["iptables" "-t" "nat" "-I" "OUTPUT" "1" "-j" "TELEPRESENCE_DNS"] : dexec.pid="7537"
2022-05-12 09:56:55.4589 info    daemon/session/dns/legacy/NAT-redirect :  : dexec.pid="7537" dexec.stream="stdin" dexec.err="EOF"
2022-05-12 09:56:55.4597 info    daemon/session/dns/legacy/NAT-redirect : finished successfully: exit status 0 : dexec.pid="7537"

connector.log

kraken@ahost ~ :( $ tail -f ~/.cache/telepresence/logs/connector.log 
2022-05-12 09:55:16.0284 info    connector/session : Server version v1.21.9-eks-0d102a7
2022-05-12 09:55:16.0284 info    connector/session : Context: aws
2022-05-12 09:55:16.0284 info    connector/session : Server: https://****.us-west-2.eks.amazonaws.com
2022-05-12 09:55:16.0285 info    connector/session : Connected to context aws (https://****.us-west-2.eks.amazonaws.com)
2022-05-12 09:55:16.6728 info    connector/session : Connecting to traffic manager...
2022-05-12 09:55:20.9147 info    connector/session : Existing Traffic Manager 2.5.8 not owned by cli or does not need upgrade, will not modify
2022-05-12 09:55:23.1476 info    connector/session : Starting additional session service reverse-conn
2022-05-12 09:55:23.1477 info    connector/session : Starting additional session service namespace-watcher
2022-05-12 09:55:23.1477 info    connector/session : Starting additional session service ingress-info-connection
2022-05-12 09:55:23.7623 info    connector/session/reverse-conn/client : Established a reverse connection to Ambassador Cloud
2022-05-12 09:56:18.9336 info    connector/session/reverse-conn:shutdown_logger : shutting down (gracefully)...
2022-05-12 09:56:18.9335 info    connector/session:shutdown_logger : shutting down (gracefully)...
2022-05-12 09:56:47.3036 info    connector/session : Connecting to k8s cluster...
2022-05-12 09:56:47.9088 info    connector/session : Server version v1.21.9-eks-0d102a7
2022-05-12 09:56:47.9089 info    connector/session : Context: aws
2022-05-12 09:56:47.9089 info    connector/session : Server: https://***.us-west-2.eks.amazonaws.com
2022-05-12 09:56:47.9090 info    connector/session : Connected to context aws (https://****.us-west-2.eks.amazonaws.com)
2022-05-12 09:56:48.6494 info    connector/session : Connecting to traffic manager...
2022-05-12 09:56:51.2642 info    connector/session : Existing Traffic Manager 2.5.8 not owned by cli or does not need upgrade, will not modify
2022-05-12 09:56:53.4017 info    connector/session : Starting additional session service reverse-conn
2022-05-12 09:56:53.4018 info    connector/session : Starting additional session service namespace-watcher
2022-05-12 09:56:53.4018 info    connector/session : Starting additional session service ingress-info-connection
2022-05-12 09:56:54.0039 info    connector/session/reverse-conn/client : Established a reverse connection to Ambassador Cloud
2022-05-12 09:59:42.2346 error   connector/server-grpc/conn=20 : Tunnel manager.Send() failed: EOF

traffic-manager pod logs

 ~ :( $ kubectl logs -f traffic-manager-7fb6b48d7d-2f4q6 -n ambassador
2022-05-09 11:00:22.3636 info    Traffic Manager v2.5.8 [pid:1]
2022-05-09 11:00:22.3855 info    Using DNS IP from kube-dns.kube-system
2022-05-09 11:00:22.3895 info    Using cluster domain "cluster.local."
2022-05-09 11:00:22.3920 info    Extracting service subnet 172.20.0.0/16 from create service error message
2022-05-09 11:00:22.3920 info    Using podCIDRStrategy: auto
2022-05-09 11:00:22.3922 info    agent-injector : Mutating webhook service is listening on :8443
2022-05-09 11:00:22.4926 info    Scanning 2 nodes
2022-05-09 11:00:22.4927 info    No subnets found
2022-05-09 11:00:22.4927 error   Unable to derive subnets from nodes
2022-05-09 11:00:22.5933 info    Deriving subnets from IPs of pods
2022-05-09 11:20:38.9014 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55636 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:38.9015 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55638 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:42.1335 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55640 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:46.4016 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55644 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:46.4019 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55642 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:49.5926 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55646 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:56.5039 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55648 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:56.5041 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55650 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:20:59.7381 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55652 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:23:41.1707 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55654 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:23:41.1707 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55656 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:23:47.6782 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55662 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:23:47.6784 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55660 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:23:49.2211 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55664 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: operation was canceled
2022-05-09 11:23:49.2212 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55664 -> 172.20.228.41:80, failed to send DialReject: rpc error: code = Unavailable desc = transport is closing
2022-05-09 11:23:50.4687 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55668 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: operation was canceled
2022-05-09 11:23:50.4688 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55668 -> 172.20.228.41:80, failed to send DialReject: rpc error: code = Unavailable desc = transport is closing
2022-05-09 11:23:52.0018 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55666 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:24:08.3103 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55670 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:24:12.5485 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55672 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:29:29.3774 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55674 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:29:40.2241 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55676 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:29:57.9021 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55678 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:30:52.3374 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55680 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:34:17.6938 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:48762 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-09 11:34:35.3234 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55682 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:34:47.1213 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55684 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:36:06.3993 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55686 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 11:36:21.5665 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:45208 -> 172.20.162.88:80, failed to establish connection: dial tcp4 172.20.162.88:80: i/o timeout
2022-05-09 11:36:25.8123 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:45210 -> 172.20.162.88:80, failed to establish connection: dial tcp4 172.20.162.88:80: i/o timeout
2022-05-09 11:54:03.9414 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55234 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:43.3375 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55236 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:43.3377 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55238 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:46.5859 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55240 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:46.5859 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55244 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:50.8619 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55246 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:50.8619 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55248 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:54.0603 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55252 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:54.0603 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55250 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:58.4437 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55256 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 11:59:58.4437 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55254 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:01.6860 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55258 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:01.6864 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55260 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:09.9536 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55264 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:09.9537 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55262 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:13.1899 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55268 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:13.1904 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55266 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:46.6456 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55272 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:46.6454 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55270 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:49.8814 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55274 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:00:49.8817 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55276 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:01:53.3239 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55280 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:01:53.3239 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55278 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:01:56.5601 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55286 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:01:56.5601 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55282 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 12:04:04.3682 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59956 -> 172.20.192.48:80, failed to establish connection: dial tcp4 172.20.192.48:80: i/o timeout
2022-05-09 12:05:47.6978 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47690 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:01.0414 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47692 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:20.3127 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47694 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:25.9483 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47696 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:38.1903 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47698 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:43.0199 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47700 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:48.7335 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47702 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:06:59.4102 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47704 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:07:05.5383 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47708 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:07:22.6897 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47712 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:07:27.5977 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47714 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:07:32.7539 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47716 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 12:08:18.7041 error   agent-injector : http: TLS handshake error from 10.0.3.59:45513: remote error: tls: unknown certificate
2022-05-09 12:08:18.7162 error   agent-injector : http: TLS handshake error from 10.0.3.59:27908: remote error: tls: unknown certificate
2022-05-09 12:08:19.7054 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 172.20.0.0:41130 -> 172.20.148.62:443 stream-to-conn loop ended because context canceled
2022-05-09 12:08:19.7490 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 172.20.0.0:41132 -> 172.20.148.62:443 stream-to-conn loop ended because context canceled
2022-05-09 12:08:22.0948 error   agent-injector : http: TLS handshake error from 10.0.3.59:54150: remote error: tls: unknown certificate
2022-05-09 12:08:22.0966 error   agent-injector : http: TLS handshake error from 10.0.3.59:61580: remote error: tls: unknown certificate
2022-05-09 12:08:23.0947 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 172.20.0.0:41134 -> 172.20.148.62:443 stream-to-conn loop ended because context canceled
2022-05-09 12:08:23.1370 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 172.20.0.0:41136 -> 172.20.148.62:443 stream-to-conn loop ended because context canceled
2022-05-09 12:10:10.2976 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 172.20.0.0:41138 -> 172.20.148.62:443 stream-to-conn loop ended because context canceled
2022-05-09 12:10:10.2977 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:50868 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-09 12:10:10.2977 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:50870 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-09 12:11:36.6011 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:50874 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-09 12:11:38.7947 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55690 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:38.7952 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55688 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:43.0322 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55696 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:43.0323 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55692 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:46.2783 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55698 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:52.7656 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55702 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:52.7656 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55700 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:11:56.0094 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55704 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:00.2825 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55706 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:00.2825 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55708 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:03.5214 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55710 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:09.0494 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55712 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:09.0495 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55714 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:12.2517 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55716 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:16.4806 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55718 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:16.4806 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55720 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:19.6821 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55724 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:27.9186 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55728 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:27.9189 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55726 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:12:31.1739 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55730 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:13:04.6374 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55732 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:13:04.6377 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55734 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:14:08.0627 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55736 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:14:08.0627 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55738 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 12:17:37.4626 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:50878 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-09 12:20:36.2854 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:50872 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-09 15:12:23.1934 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55740 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:12:52.9105 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55742 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:13:11.7197 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55744 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:13:44.8054 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55288 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:44.8056 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55290 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:48.0416 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55294 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:48.0416 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55292 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:52.2746 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55296 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:52.2749 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55298 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:55.4693 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55300 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:13:55.4693 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55302 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:03.7219 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55306 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:03.7221 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55304 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:06.9544 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55308 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:06.9544 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55310 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:40.1631 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55312 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:40.1632 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55314 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:43.3565 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55318 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:43.3565 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55316 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:48.2314 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55320 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:48.2318 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55322 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:51.4664 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55324 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:14:51.6108 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55326 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:15:24.6984 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55748 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:24.6988 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55746 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:28.9617 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55750 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:28.9618 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55752 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:32.1974 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55754 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:44.1215 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55756 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:44.1216 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55758 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:15:47.3533 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55760 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:16:20.5624 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55762 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:16:20.5624 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55764 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:16:23.7944 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55766 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:17:27.2334 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55768 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:17:27.2335 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55770 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:20:18.5538 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55772 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:20:18.5538 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55774 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:20:34.5254 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55328 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:20:34.5256 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55332 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:20:37.7168 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55334 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:20:37.7168 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55336 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-09 15:21:32.6623 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55776 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:21:47.9861 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55778 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:24:07.7217 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:55780 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-09 15:29:29.6939 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47740 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 15:29:35.3981 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47742 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-09 15:29:41.7610 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:47744 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:30:52.3444 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:54064 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-10 12:31:50.9534 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 10.0.0.0:45692 -> 10.0.3.252:80, failed to establish connection: dial tcp4 10.0.3.252:80: connect: connection refused
2022-05-10 12:32:27.2218 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 10.0.0.0:45694 -> 10.0.3.252:80, failed to establish connection: dial tcp4 10.0.3.252:80: connect: connection refused
2022-05-10 12:32:31.8958 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:51548 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-10 12:32:52.8977 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:60636 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-10 12:33:05.6734 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59230 -> 172.20.185.56:30213, failed to establish connection: dial tcp4 172.20.185.56:30213: i/o timeout
2022-05-10 12:33:13.0940 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:60638 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-10 12:33:15.4089 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:51544 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-10 12:33:52.0213 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:54066 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-10 12:34:12.4413 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:54070 -> 172.20.228.41:80, failed to establish connection: dial tcp4 172.20.228.41:80: i/o timeout
2022-05-10 12:34:55.3693 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:40952 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:35:07.9078 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:40954 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:35:15.4093 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:40956 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:35:21.0615 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:40958 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:35:50.1466 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:40960 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-10 12:37:09.4836 error   httpd/conn=127.0.0.1:8081 :    CONN tcp 10.0.0.0:51542 -> 10.0.3.252:8081 stream-to-conn loop ended because context canceled
2022-05-11 11:04:49.0220 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:45634 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-11 11:05:03.0774 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59088 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:05:41.8213 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51098 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:06:49.2985 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51100 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:06:56.6050 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:45636 -> 172.20.98.4:443, failed to establish connection: dial tcp4 172.20.98.4:443: i/o timeout
2022-05-11 11:07:19.4777 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51102 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:08:26.3145 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51104 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:09:29.9083 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51106 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:09:38.8136 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51108 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:12:34.9430 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51110 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:12:58.5455 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51114 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:12:58.5456 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51112 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:01.7808 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51118 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:06.0706 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51120 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:06.0706 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51122 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:09.2635 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51124 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:17.4767 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51126 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:17.4767 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51128 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:20.7097 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51130 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:53.9145 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51134 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:53.9145 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51132 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:13:57.1087 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51138 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:15:00.5500 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51140 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:15:00.5501 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:51142 -> 172.20.98.4:80, failed to establish connection: dial tcp4 172.20.98.4:80: i/o timeout
2022-05-11 11:15:50.5894 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59092 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:15:50.5894 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59090 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:15:53.8257 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59094 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:15:58.0862 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59096 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:15:58.0864 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59098 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:01.3790 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59100 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:09.5848 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59104 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:09.5852 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59102 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:12.8220 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59106 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:51.6825 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59108 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:16:52.6823 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59110 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:17:57.7677 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59114 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:17:57.7677 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59112 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:41.9395 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59116 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:41.9397 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59118 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:47.3463 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59120 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:47.3463 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59124 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:50.5440 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59126 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:55.0617 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59130 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:55.0622 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59132 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:40:58.2939 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59134 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:46:09.0173 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59138 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:46:09.0175 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59140 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:53:44.7554 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59144 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-11 11:53:44.7556 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59142 -> 172.20.185.56:80, failed to establish connection: dial tcp4 172.20.185.56:80: i/o timeout
2022-05-12 07:59:05.6335 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59760 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-12 07:59:17.0121 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59762 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout
2022-05-12 07:59:27.1460 error   httpd/conn=127.0.0.1:8081 : !! CONN tcp 172.20.0.0:59764 -> 172.20.184.94:6379, failed to establish connection: dial tcp4 172.20.184.94:6379: i/o timeout

I think there is something with AWS security groups, so I need to allow some traffic and open specific ports but I don't know exactly

May 12 '22 08:05 SunPj

Adding AWS Security Groups doc page for reference: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html

May 13 '22 21:05 cindymullins-dw

It's definitely not an SG issue. Working on this a bit with your team, we discovered that in EKS the traffic was making it's way to the service in the cluster and the service was responding. However the traffic wasn't making their way back down to the local machine.

May 16 '22 16:05 zquintana

It's likely this has been improved or resolved in more recent releases. Please reopen if persisting on newer versions.

Mar 19 '24 20:03 cindymullins-dw

telepresence telepresence copied to clipboard

Unable to reach services in cluster

telepresence
telepresence copied to clipboard