caddyserver
Proxied websocket connexion systematically closed after ~9s
Caddy version: 2.9.1 (also 2.8.10 did not try before)
Hi,
TLDR: A proxied websocket connection through caddy gets closed at ~9-10s. Caddy config has all timeouts set to high values. I observe the same thing with proxied uploads to nextcloud.
Details:
I am using caddy as a proxy to my homelab. The upstreams are reachable through wireguard. I proxy dozens of services but I am observing this ~10s timeout issue with 2 services (tty-share websocket based service, nextcloud uploads).
I don't know if the phenomenon with both services is a coincidence so I will focus on the Websocket problem.
The service is a tty sharing app tty-share, it relies on a tty-proxy service
which is TCP-TLS tunneled through caddy-l4. The tty-sharing websocket works flawlessly when connected to directly without Caddy or proxy service. However when proxied through caddy, the connection drops at around 9s sometimes 10s.
There is no firewall blocking or dropping the connection on the upstream side. The tty-proxy service does not log any error except that the connection got closed.
Caddy config:
},
"apps": {
"http": {
"servers": {
"srv0": {
"idle_timeout": 600000000000,
"listen": [
":443"
],
"logs": {"logger_names":…},
"read_header_timeout": 30000000000,
"read_timeout": 120000000000,
"routes": […],
"write_timeout": 300000000000
},
"srv1": {
"idle_timeout": 600000000000,
"listen": [
":80"
],
"read_header_timeout": 30000000000,
"read_timeout": 120000000000,
"routes": […],
"write_timeout": 300000000000
}
}
}
{"handle":…},
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"stream_timeout": 3600000000000,
"transport": {
"dial_timeout": 3600000000000,
"protocol": "http",
"read_timeout": 3600000000000,
"write_timeout": 3600000000000
},
"upstreams": [
{
"dial": "tty-proxy.upstream:8080"
}
]
}
]
}
]
}
],
"match": [
{
"host": [
"tty.public.host"
]
}
],
"terminal": true
},
{"handle":…},
Caddy log for the connection session:
caddy-1 | {"level":"debug","ts":1744809297.4248097,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"tty-proxy.upstream:8080","duration":0.221202952,"request":{"remote_ip":"172.18.0.5","remote_port":"54028","client_ip":"172.18.0.5","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/YSj1-ho0M6xhR1vl-6AKDgRXhYEKwxN1lG4oJ9QuIYPj-IHqnilHq9JbLzDBysbK3kg/ws","headers":{"Pragma":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["my.host"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["172.18.0.5"],"Connection":["Upgrade"],"Sec-WebSocket-Key":["5jPkmgijMAtp/Ylw2FsQNQ=="],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"Origin":["https://my.host"],"X-Forwarded-Proto":["https"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"headers":{"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Accept":["GsAeD21Zyv5kyu+93//2xh7TQ2w="]},"status":101}
caddy-1 | {"level":"debug","ts":1744809297.4248843,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"tty-proxy.upstream:8080","duration":0.221202952,"request":{"remote_ip":"172.18.0.5","remote_port":"54028","client_ip":"172.18.0.5","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/YSj1-ho0M6xhR1vl-6AKDgRXhYEKwxN1lG4oJ9QuIYPj-IHqnilHq9JbLzDBysbK3kg/ws","headers":{"Pragma":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["my.host"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["172.18.0.5"],"Connection":["Upgrade"],"Sec-WebSocket-Key":["5jPkmgijMAtp/Ylw2FsQNQ=="],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"Origin":["https://my.host"],"X-Forwarded-Proto":["https"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"http_version":2}
caddy-1 | {"level":"debug","ts":1744809305.503886,"logger":"http.handlers.reverse_proxy","msg":"streaming error","upstream":"tty-proxy.upstream:8080","duration":0.221202952,"request":{"remote_ip":"172.18.0.5","remote_port":"54028","client_ip":"172.18.0.5","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/YSj1-ho0M6xhR1vl-6AKDgRXhYEKwxN1lG4oJ9QuIYPj-IHqnilHq9JbLzDBysbK3kg/ws","headers":{"Pragma":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["my.host"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["172.18.0.5"],"Connection":["Upgrade"],"Sec-WebSocket-Key":["5jPkmgijMAtp/Ylw2FsQNQ=="],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"Origin":["https://my.host"],"X-Forwarded-Proto":["https"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"error":"client disconnected"}
caddy-1 | {"level":"debug","ts":1744809305.503926,"logger":"http.handlers.reverse_proxy","msg":"connection closed","upstream":"tty-proxy.upstream:8080","duration":0.221202952,"request":{"remote_ip":"172.18.0.5","remote_port":"54028","client_ip":"172.18.0.5","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/YSj1-ho0M6xhR1vl-6AKDgRXhYEKwxN1lG4oJ9QuIYPj-IHqnilHq9JbLzDBysbK3kg/ws","headers":{"Pragma":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["my.host"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["172.18.0.5"],"Connection":["Upgrade"],"Sec-WebSocket-Key":["5jPkmgijMAtp/Ylw2FsQNQ=="],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"Origin":["https://my.host"],"X-Forwarded-Proto":["https"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"duration":8.078935445}
Chrome network log:
Nextcloud Uploads
As I said above this might be a coincidence so I am refereing to this issue if it helps identify the problem.
When I ever I start an UPLOAD session with my proxied nextcloud instance the upload fails at around 9-10s. This is consistent no matter the size of file.
Hello, I have a similar problem with version 2.9.1 for Home Assistant behind caddy as a proxy. Everything worked well with version 2.8.4, but many add-ons for Home Assistant (like Zigbee2MQTT and Terminal which use ... ttyd!!) that rely on web-socket have exhibited problems (like disconnections for Terminal) since the last stable version (2.9.1) of caddy. I had to rollback the version and no more problem.
@rphlrmblt thanks for the remark, I remember not having this problem before than I thought it was a nextcloud regression but now I feel it is definitely caddy. I will downgrade and update the comment if it resolves my issue.
@blob42 @rphlrmblt Can you try build caddy using the latest source, xcaddy build master to see if this issue persists? I suspect it's due to websocket over h2 compatiblity, which is disabled in the master.
I observed a similar behaviour with version 2.9.1 and the Uptime Kuma 2 beta - regular connection closing, disconnects, and reconnects. I’ve just tested the current master branch and it works fine on my end.
I've build caddy using the latest source, after build xcaddy and launched xcaddy build master :
2025/04/18 12:26:05 [INFO] exec (timeout=0s): /usr/bin/go build -o /home/rphlrmblt/bin/xc/caddy -ldflags -w -s -trimpath -tags nobadger,nomysql,nopgx
2025/04/18 12:27:02 [INFO] Build complete: ./caddy
2025/04/18 12:27:02 [INFO] Cleaning up temporary folder: /tmp/buildenv_2025-04-18-1225.760567498
././caddy version
v2.10.0-beta.4.0.20250417224306-35c8c2d92d26 h1:eIhu5DteQptnAvHDgbbrDxUu4rnw/3H/sKDgzVN5ltU=
Renamed the rollbacked caddy v2.8.4 executable and replaced with the new one in /usr/bin/
Restart the service with a systemctl restart caddy and .... no more problem with HomeAssistant plugins seen with the version 2.9.1, Terminal work flawlessly !
So it seems to be "fixed" in master branch as expected by @WeidiDeng
@WeidiDeng Sorry I didn't have time to update ticket. I am still observing the same issue with 2.10/master version on websockets or nextcloud uploads. I suspect this is unrelated to the problem adressed by other users. Is there anything I can do to help debug this issue ? I can share packet captures.
Yes here is the debug log for a websocket disconnecting session. I used the same caddy conf as on my initial post
> docker compose exec caddy caddy version
v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=
Built with xcaddy using the docker image caddy:2.10-builder
caddy-1 | {"level":"debug","ts":1746019101.9487467,"logger":"events","msg":"event","name":"tls_get_certificate","id":"48963774-fdc2-40c8-b245-7baac479ccac","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"my.host","SupportedCurves":[4588,29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"172.18.0.15","Port":46530,"Zone":""},"LocalAddr":{"IP":"172.18.42.42","Port":443,"Zone":""}}}}
caddy-1 | {"level":"debug","ts":1746019101.9490511,"logger":"tls.handshake","msg":"choosing certificate","identifier":"my.host","num_choices":1}
caddy-1 | {"level":"debug","ts":1746019101.949219,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"my.host","subjects":["my.host"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"65c0cc9f04e736b1c2181262d20d8ba94d54c89d90c5e3f5e5cdc89fc3284d94"}
caddy-1 | {"level":"debug","ts":1746019101.9496937,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.15","remote_port":"46530","subjects":["my.host"],"managed":true,"expiration":1750764303,"hash":"65c0cc9f04e736b1c2181262d20d8ba94d54c89d90c5e3f5e5cdc89fc3284d94"}
caddy-1 | {"level":"debug","ts":1746019102.023451,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"tty-proxy.upstream:8080","total_upstreams":1}
caddy-1 | {"level":"debug","ts":1746019102.16661,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"tty-proxy.upstream:8080","duration":0.142872674,"request":{"remote_ip":"172.18.0.15","remote_port":"46530","client_ip":"172.18.0.15","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Mode":["navigate"],"Te":["trailers"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=0, i"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["none"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["172.18.0.15"],"X-Forwarded-Host":["my.host"],"Via":["2.0 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"headers":{"Ttyshare-Tty-Wspath":["/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws"],"Ttyshare-Tunnel-Wspath":["/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/tws"],"Ttyshare-Version":["2"],"Ttyshare-Wspath":["/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws"],"Date":["Wed, 30 Apr 2025 13:18:22 GMT"],"Content-Length":["651"],"Content-Type":["text/html; charset=utf-8"]},"status":200}
caddy-1 | {"level":"debug","ts":1746019104.4146662,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"tty-proxy.upstream:8080","total_upstreams":1}
caddy-1 | {"level":"debug","ts":1746019104.7205033,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"tty-proxy.upstream:8080","duration":0.30577301,"request":{"remote_ip":"172.18.0.15","remote_port":"46530","client_ip":"172.18.0.15","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/static/tty-share.js","headers":{"Sec-Fetch-Mode":["no-cors"],"Dnt":["1"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["my.host"],"Via":["2.0 Caddy"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Gpc":["1"],"Referer":["https://my.host/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/"],"Accept":["*/*"],"X-Forwarded-For":["172.18.0.15"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["script"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"headers":{"Content-Type":["text/javascript; charset=utf-8"],"Date":["Wed, 30 Apr 2025 13:18:24 GMT"]},"status":200}
caddy-1 | {"level":"debug","ts":1746019105.7112463,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"tty-proxy.upstream:8080","total_upstreams":1}
caddy-1 | {"level":"debug","ts":1746019105.7375498,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"tty-proxy.upstream:8080","total_upstreams":1}
caddy-1 | {"level":"debug","ts":1746019105.8656838,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"tty-proxy.upstream:8080","duration":0.128035272,"request":{"remote_ip":"172.18.0.15","remote_port":"46530","client_ip":"172.18.0.15","proto":"HTTP/2.0","method":"GET","host":"my.host","uri":"/favicon.ico","headers":{"Sec-Fetch-Site":["same-origin"],"Priority":["u=6"],"Via":["2.0 Caddy"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"X-Forwarded-For":["172.18.0.15"],"X-Forwarded-Proto":["https"],"Referer":["https://my.host/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/"],"Sec-Fetch-Mode":["no-cors"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Dnt":["1"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["image"],"X-Forwarded-Host":["my.host"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Te":["trailers"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.host"}},"headers":{"Date":["Wed, 30 Apr 2025 13:18:25 GMT"],"Content-Length":["587"],"Content-Type":["text/html; charset=utf-8"]},"status":200}
caddy-1 | {"level":"debug","ts":1746019105.9297833,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"tty-proxy.upstream:8080","duration":0.218454672,"request":{"remote_ip":"172.18.0.15","remote_port":"34360","client_ip":"172.18.0.15","proto":"HTTP/1.1","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws","headers":{"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["my.host"],"Sec-WebSocket-Key":["nRVVs0Dp8oONZW+K9gnVtw=="],"Upgrade":["websocket"],"Sec-Gpc":["1"],"Accept":["*/*"],"Sec-Fetch-Mode":["websocket"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Cache-Control":["no-cache"],"Connection":["Upgrade"],"Sec-Fetch-Dest":["empty"],"Pragma":["no-cache"],"Sec-WebSocket-Extensions":["permessage-deflate"],"Origin":["https://my.host"],"Sec-WebSocket-Version":["13"],"X-Forwarded-For":["172.18.0.15"],"Via":["1.1 Caddy"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"my.host"}},"headers":{"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Accept":["ZIlfsSuoPvK2ySAk9unSDDp9F20="]},"status":101}
caddy-1 | {"level":"debug","ts":1746019105.930273,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"tty-proxy.upstream:8080","duration":0.218454672,"request":{"remote_ip":"172.18.0.15","remote_port":"34360","client_ip":"172.18.0.15","proto":"HTTP/1.1","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws","headers":{"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["my.host"],"Sec-WebSocket-Key":["nRVVs0Dp8oONZW+K9gnVtw=="],"Upgrade":["websocket"],"Sec-Gpc":["1"],"Accept":["*/*"],"Sec-Fetch-Mode":["websocket"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Cache-Control":["no-cache"],"Connection":["Upgrade"],"Sec-Fetch-Dest":["empty"],"Pragma":["no-cache"],"Sec-WebSocket-Extensions":["permessage-deflate"],"Origin":["https://my.host"],"Sec-WebSocket-Version":["13"],"X-Forwarded-For":["172.18.0.15"],"Via":["1.1 Caddy"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"my.host"}},"http_version":1}
caddy-1 | {"level":"debug","ts":1746019115.6581,"logger":"http.handlers.reverse_proxy","msg":"streaming error","upstream":"tty-proxy.upstream:8080","duration":0.218454672,"request":{"remote_ip":"172.18.0.15","remote_port":"34360","client_ip":"172.18.0.15","proto":"HTTP/1.1","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws","headers":{"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["my.host"],"Sec-WebSocket-Key":["nRVVs0Dp8oONZW+K9gnVtw=="],"Upgrade":["websocket"],"Sec-Gpc":["1"],"Accept":["*/*"],"Sec-Fetch-Mode":["websocket"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Cache-Control":["no-cache"],"Connection":["Upgrade"],"Sec-Fetch-Dest":["empty"],"Pragma":["no-cache"],"Sec-WebSocket-Extensions":["permessage-deflate"],"Origin":["https://my.host"],"Sec-WebSocket-Version":["13"],"X-Forwarded-For":["172.18.0.15"],"Via":["1.1 Caddy"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"my.host"}}}
caddy-1 | {"level":"debug","ts":1746019115.6593914,"logger":"http.handlers.reverse_proxy","msg":"connection closed","upstream":"tty-proxy.upstream:8080","duration":0.218454672,"request":{"remote_ip":"172.18.0.15","remote_port":"34360","client_ip":"172.18.0.15","proto":"HTTP/1.1","method":"GET","host":"my.host","uri":"/s/0vFYsHIdRjnApw3FleAvRiEgnA4sYLJMy_x2w0f_2QvTx-ama7p4VPwumx4jSwRgOFY/ws","headers":{"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["my.host"],"Sec-WebSocket-Key":["nRVVs0Dp8oONZW+K9gnVtw=="],"Upgrade":["websocket"],"Sec-Gpc":["1"],"Accept":["*/*"],"Sec-Fetch-Mode":["websocket"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Cache-Control":["no-cache"],"Connection":["Upgrade"],"Sec-Fetch-Dest":["empty"],"Pragma":["no-cache"],"Sec-WebSocket-Extensions":["permessage-deflate"],"Origin":["https://my.host"],"Sec-WebSocket-Version":["13"],"X-Forwarded-For":["172.18.0.15"],"Via":["1.1 Caddy"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"my.host"}},"duration":9.728091938}
Regarding proxied uploads to nextcloud also dropping around the same period I want to replicate this issue with a minimal setup and a simple upload app upstream. I still don't know if proxying through wireguard has anything to do with this, it would be complicated to replicate the same environment without wireguard.
EDIT: Is there some more debug logging I could do on a custom caddy build to troubleshoot this issue ?
@blob42 Can you try building caddy using golang 1.23 instead of golang 1.24? Maybe golang changed something regarding underlying sockets.
@WeidiDeng Ok I will try but I am starting to suspect this is a network problem with my homelab, the server is behind a switch that seems to get congested sometimes. I will update this issue after some more testing.
Also seeing this on caddy 2.8 and 2.10. Latest built with 1.24. It's not something I can test with 1.23 without quite a bit of effort. I'm using caddy with home-assistant and its websockets are silently dropping. Basic network tests don't show any issues at that level for me.
Logged at debug and there's nothing apart form the typical "request handled" info entry. All default settings in caddy.
Confirming that's not a network issue. Moved to nginx and it "just works" with no timeouts/dropouts.
@viraptor Does this issue happen on caddy 2.7.*? What's the last version that worked correctly for you?
@blob42 Did you notice this problem on older caddy version? What's the last version that worked correctly for you?
How did you configure nextcloud to use websockets for uploads? I deployed nextcloud myself to see if there is any error when uploading but no websockets are involved.
I don't use tty-share, though I use ttyd with a similar purpose and haven't encountered any issue.
haven't forgotten the request - I'll do the testing when I have more time.
@blob42 Did you notice this problem on older caddy version? What's the last version that worked correctly for you?
How did you configure nextcloud to use websockets for uploads? I deployed nextcloud myself to see if there is any error when uploading but no websockets are involved.
I don't use tty-share, though I use ttyd with a similar purpose and haven't encountered any issue.
Sorry I don't remember the last version that worked but it was around 2.8 something.
Regarding Nextcloud the problem could also be from nextcloud itself and there are no websockets involved indeed. I only mentioned Nextcloud because I noticed that the upload is dropped at around 10s mark exactly like the websocket issue. I used a timer and when I upload big files the connection drops always around the same 10s mark no matter the file size. ( I will also test with throttling to validate that it is time related and not file size)
Back to websockets which is the topic of this issue, the 2.10 patch did not resolve my websocket dropping with tty-share.
I did some tests outside my "problematic" network segment and can still observe the drops. Still seriously planning to do a proper full pcap capture from multiple sides and dig down.
What is left to test is:
- Directly export nextcloud ports and bypass caddy to validate whether the issue is from Caddy.
- Try git bisect from ( find a working commit)
@WeidiDeng I managed to fix the issue and it had nothing to do with Caddy. It was a encrypted-dns with DoH proxy in front of caddy where all TLS traffic passed through. I really feel stupid 🤦 to not have noticed it earlier and I forgot I had it installed. I realized that was the issue when experimenting with non https traffic.
Nextcloud and tty-share do not drop connexions any more. I tried very long websocket sessions and I did not observe any drops.
Feel free to close the issue or keep it open for @viraptor
Thanks for the follow-up, @blob42 .
Indeed, I'm inclined to think this is something external to Caddy.
We can reopen if @viraptor is able to confirm otherwise. Thanks!
I don't understand why a DoH proxy would affect websockets? What's your setup here, exactly? What do DNS queries have to do with websocket connections? Those should be separate. Why are you putting a DoH proxy in front of HTTP services?
Glad you found the issue though.
@francislavoie Sorry for the confusing comment. I had encrypted-dns catching all traffic on 443 and proxying tls traffic back to caddy and DoH traffic to doh-server.
I see, so the problem was with https://github.com/DNSCrypt/encrypted-dns-server 's proxying?
I observed a similar behaviour with version 2.9.1 and the Uptime Kuma 2 beta - regular connection closing, disconnects, and reconnects. I’ve just tested the current master branch and it works fine on my end.
I'm having the same problem using caddy-docker-proxy 2.10 and Uptima Kuma 2.0.2. Were you able to solve this?
