inlets-operator
inlets-operator copied to clipboard
[Bug] inlets client failed to connect when exposing port is 8080
Expected Behaviour
The inlets client in k8s cluster should connect to inlets server successfully.
Current Behaviour
The inlets client in k8s cluster fails to connect inlets server.
Note that if we use inlets client outside k8s cluster, the connection will succeed, meaning the inlets server is working just fine.
Possible Solution
Steps to Reproduce (for bugs)
kind create cluster
# Install inlets-operator
kubectl apply -f ./artifacts/crds/
kubectl create secret generic inlets-access-key --from-literal inlets-access-key=<Linode Access Token>
helm install --set provider=linode,region=us-east inlets-operator ./chart/inlets-operator
# Install and expose echoserver
kubectl run echoserver --image=gcr.io/google_containers/echoserver:1.4 --port=8080
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
labels:
run: echoserver
name: echoserver
namespace: default
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
run: echoserver
type: LoadBalancer
EOF
Logs
kubectl logs deployment.apps/echoserver-tunnel-client
2020/07/18 10:25:36 Welcome to inlets.dev! Find out more at https://github.com/inlets/inlets
2020/07/18 10:25:36 Starting client - version 2.7.3
2020/07/18 10:25:36 Upstream: => http://echoserver:8080
2020/07/18 10:25:36 Token: "p9DzT5gtqubn0EefdK9f9IxgEXjRGAXx57tiEnwR2henRhu7XovASoOvcHhklq0i"
time="2020/07/18 10:25:36" level=info msg="Connecting to proxy" url="ws://45.79.148.100:8080/tunnel"
time="2020/07/18 10:25:36" level=error msg="Failed to connect to proxy. Response status: 200 - 200 OK. Response body: CLIENT VALUES:\nclient_address=10.244.0.1\ncommand=GET\nreal path=/tunnel\nquery=nil\nrequest_version=1.1\nrequest_uri=http://45.79.148.100:8080/tunnel\n\nSERVER VALUES:\nserver_version=nginx: 1.10.0 - lua: 10001\n\nHEADERS RECEIVED:\nauthorization=Bearer p9DzT5gtqubn0EefdK9f9IxgEXjRGAXx57tiEnwR2henRhu7XovASoOvcHhklq0i\nconnection=Upgrade\nhost=45.79.148.100:8080\nsec-websocket-key=b5J+fZqRtaHdOHjv3hZ/7w==\nsec-websocket-version=13\nupgrade=websocket\nuser-agent=Go-http-client/1.1\nx-inlets-id=dcb2f0ffac964928a06bd64aaf5d60e8\nx-inlets-upstream==http://echoserver:8080\nBODY:\n-no body in request-" error="websocket: bad handshake"
Context
Your Environment
- Can be reproduced in different k8s environments: Linode, DigitalOcean, kinD.
- Can be reproduced using different apps, e.g. echoserver, openfaas
- Can only be reproduced with free version of inlets, unable to reproduce with inlets PRO
The root cause may be that iptables rules are set up against port 8080
. These rules can be seen by logging into k8s nodes.
As we can see below, the 45.79.148.100:8080 will eventually go to 10.244.0.6:8080 (which is the application pod)
root@kind-control-plane:/# iptables-save | grep -E '4AM65LVENFDJEVZH|M3SSESRYD7TVHMEI|8080'
:KUBE-FW-M3SSESRYD7TVHMEI - [0:0]
:KUBE-SEP-4AM65LVENFDJEVZH - [0:0]
:KUBE-SVC-M3SSESRYD7TVHMEI - [0:0]
-A KUBE-FW-M3SSESRYD7TVHMEI -m comment --comment "default/echoserver:http loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-M3SSESRYD7TVHMEI -m comment --comment "default/echoserver:http loadbalancer IP" -j KUBE-SVC-M3SSESRYD7TVHMEI
-A KUBE-FW-M3SSESRYD7TVHMEI -m comment --comment "default/echoserver:http loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/echoserver:http" -m tcp --dport 32306 -j KUBE-SVC-M3SSESRYD7TVHMEI
-A KUBE-SEP-4AM65LVENFDJEVZH -s 10.244.0.6/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-4AM65LVENFDJEVZH -p tcp -m tcp -j DNAT --to-destination 10.244.0.6:8080
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.76.33/32 -p tcp -m comment --comment "default/echoserver:http cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.76.33/32 -p tcp -m comment --comment "default/echoserver:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-M3SSESRYD7TVHMEI
-A KUBE-SERVICES -d 45.79.148.100/32 -p tcp -m comment --comment "default/echoserver:http external IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 45.79.148.100/32 -p tcp -m comment --comment "default/echoserver:http external IP" -m tcp --dport 8080 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-M3SSESRYD7TVHMEI
-A KUBE-SERVICES -d 45.79.148.100/32 -p tcp -m comment --comment "default/echoserver:http external IP" -m tcp --dport 8080 -m addrtype --dst-type LOCAL -j KUBE-SVC-M3SSESRYD7TVHMEI
-A KUBE-SERVICES -d 45.79.148.100/32 -p tcp -m comment --comment "default/echoserver:http loadbalancer IP" -m tcp --dport 8080 -j KUBE-FW-M3SSESRYD7TVHMEI
-A KUBE-SVC-M3SSESRYD7TVHMEI -j KUBE-SEP-4AM65LVENFDJEVZH
The resources and their IP addresses:
kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/echoserver-5d9b4777bc-bzhn4 1/1 Running 0 61m 10.244.0.6 kind-control-plane <none> <none>
pod/echoserver-tunnel-client-5d8b8dc45f-6pz8t 1/1 Running 0 32m 10.244.0.11 kind-control-plane <none> <none>
pod/inlets-operator-58558dd5cc-njxc5 1/1 Running 0 59m 10.244.0.7 kind-control-plane <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/echoserver LoadBalancer 10.96.76.33 45.79.148.100,45.79.148.100 8080:32306/TCP 33m run=echoserver
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 69m <none>
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/echoserver 1/1 1 1 61m echoserver gcr.io/google_containers/echoserver:1.4 run=echoserver
deployment.apps/echoserver-tunnel-client 1/1 1 1 32m client inlets/inlets:2.7.3 app.kubernetes.io/name=echoserver-tunnel-client
deployment.apps/inlets-operator 1/1 1 1 59m inlets-operator inlets-operator:latest app.kubernetes.io/instance=inlets-operator,app.kubernetes.io/name=inlets-operator
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/echoserver-5d9b4777bc 1 1 1 61m echoserver gcr.io/google_containers/echoserver:1.4 pod-template-hash=5d9b4777bc,run=echoserver
replicaset.apps/echoserver-tunnel-client-5d8b8dc45f 1 1 1 32m client inlets/inlets:2.7.3 app.kubernetes.io/name=echoserver-tunnel-client,pod-template-hash=5d8b8dc45f
replicaset.apps/inlets-operator-58558dd5cc 1 1 1 59m inlets-operator inlets-operator:latest app.kubernetes.io/instance=inlets-operator,app.kubernetes.io/name=inlets-operator,pod-template-hash=58558dd5cc
Pinging @mauilion for ideas
The server is running on a VM with a remote IP. It accepts traffic on port 80, then uses its websocket to talk to the inlets client (a Deployment on the local Kubernetes cluster)
The "upstream" URL for the inlets client is like the upstream value of a reverse proxy, so it will send any requests received from the server there.
The issue appears to be before we get that far, with the client trying to do a HTTP connection to ws://45.79.148.100:8080/tunnel
- instead of hitting the external VM's IP on port 8080, it gets booted over to the service "echoserver" which also listens on port 8080.