cloudflared
cloudflared copied to clipboard
Published
20 hours ago •
cloudflare
cloudflare
Tunnel stops working
Describe the bug The tunnel randomly stops working
Environment and versions Docker image cloudflare/cloudflared:2022.2.0
Logs and errors
2022-03-04T08:21:22Z INF Connection cfa57569-7451-44ef-aab4-5ee99096d6df registered connIndex=2 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=1
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=1
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=1
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=3
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=3
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=3 2022-03-04T08:38:11Z INF Connection ac97da84-de5a-48b1-8adb-1aa61bac2386 registered connIndex=1 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=2
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=2
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=2
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=2
Recover from http2 response writer panic, error goroutine 943226123 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f
panic({0xe8c700, 0x11bd190})
/usr/local/go/src/runtime/panic.go:1038 +0x215
golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67c061a98, 0x0}, {0x0, 0x7ff6a2eb4f18})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150
golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0xdf})
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64
zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149
github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}}) /go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138 zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3
created by zombiezen.com/go/capnproto2/rpc.NewConn
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d
Recover from http2 response writer panic, error goroutine 943226123 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f
panic({0xe8c700, 0x11bd190})
/usr/local/go/src/runtime/panic.go:1038 +0x215
golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67aee25d0, 0x0}, {0x0, 0x7ff6a2eb4f18})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150
golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0x35})
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64
zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149
github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138
zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3
created by zombiezen.com/go/capnproto2/rpc.NewConn
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=0
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=0
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=0
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=0
2022-03-04T08:38:12Z INF Connection 2a244dc7-b8e0-4342-a75f-9117c6130b23 registered connIndex=3 location=IAD
2022-03-04T08:38:12Z INF Connection f616308f-6ca3-43e8-a599-c5ce3dea3f5a registered connIndex=2 location=IAD
2022-03-04T08:38:12Z INF Connection 7c2787a1-f967-4d7e-875c-f27d39974d5f registered connIndex=0 location=IAD
2022-03-04T08:38:31Z INF Lost connection with the edge connIndex=3
2022-03-04T08:38:31Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:38:31Z INF Retrying connection in up to 1s seconds connIndex=3
2022-03-04T08:38:31Z INF Unregistered tunnel connection connIndex=3
2022-03-04T08:38:32Z INF Connection 686d1572-ddd6-4316-9b41-ee6ce6d54e6b registered connIndex=3 location=IAD
2022-03-04T08:40:01Z INF Lost connection with the edge connIndex=3
2022-03-04T08:40:01Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:40:01Z INF Retrying connection in up to 1s seconds connIndex=3
Additional context Running on Kubernetes 1.19.11 Docker version 20.10.7, build 20.10.7-0ubuntu5~18.04.3
I am not sure if this is related, but we are having problems with cloudflared and Kubernetes (DOKS) as well:
Pod on kubernetes is creating tcp tunnel (running cloudflared access tcp..) to Windows host (running cloudflared tunnel ...).
It works fine about 30 seconds after start and then it starts getting errors:
2022-03-04T11:57:01Z ERR failed to connect to origin error="websocket: bad handshake" originURL=<REDACTED>
From client side this errors with ECONNRESET.
About 30-40% of connections succeeds.
Again, not sure if related at all.
At the same time another (windows) machine running cloudflared access tcp.. to same endpoint works 100%.
Running Kubernetes 1.20.15-do.0.
