Websocket+HTTP/2 error with cloudflare load balancing

[maartenbreddels]

I didn't find much online about this, but when connecting through cloudflare to a tornado WebSocket handler I received these headers:

Host: dataframe.vaex.io
X-Real-Ip: 141.101.105.77
X-Forwarded-For: 89.188.19.102, 141.101.105.77
Connection: upgrade
Accept-Encoding: gzip
Cf-Ipcountry: NL
Cf-Ray: 5afa07af2b3cbd82-AMS
X-Forwarded-Proto: https
Cf-Visitor: {"scheme":"https"}
User-Agent: curl/7.68.0
Accept: */*
Origin: http://dataframe.vaex.io
Cf-Request-Id: 03d02321780000bd82a2118200000001
Cf-Connecting-Ip: 89.188.19.102
Cdn-Loop: cloudflare

After which you get the error Connection" must be "Upgrade". The stacktrace in tornado is pretty vague (self.client is None at https://github.com/tornadoweb/tornado/blob/ad6e4acf9eb0df6a07f48fddd5945510bcd4a555/tornado/simple_httpclient.py#L658).

But using curl helped:

$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: dataframe.vaex.io" -H "Origin: http://dataframe.vaex.io" https://dataframe.vaex.io/websocket

Note the https

https://community.cloudflare.com/t/websocket-pass-through-crashes-worker-script/78482/7 gave me a hint to disable http2:

$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: dataframe.vaex.io" -H "Origin: http://dataframe.vaex.io" https://dataframe.vaex.io/websocket --http1.1  

This worked 🎉

I didn't see an option to choose the HTTP version in tornado, so i disabled https in cloudflare, and I now don't have an issue.

Sorry for the low quality hasty, report. I am in a hurry, but it might be useful for the author(s) or someone with similar issues.

[ploxiln]

Tornado does not support http/2 (unless you use the separate add-on https://github.com/bdarnell/tornado_http2). So perhaps when Cloudflare translates http/2 from the client to http/1.1 to the tornado backend, it changes "Upgrade" to "upgrade". Or maybe a browser using http/2 sends the header differently. I'm not sure if the value is supposed to be case-insensitive.

Note that http/2 from browsers requires https, so switching to plain http is one way to force http/1.1 (and directly forcing http/1.1 with a curl flag is another way).

Note also that mixing websockets with http/2 is not trivial, since websockets normally take-over the entire http/1.1 connection, while http/2 can have other requests going on the same connection, concurrently. (See https://medium.com/@pgjones/http-2-websockets-81ae3aab36dd) So maybe translating from http/2 websocket to http/1.1 websocket in the cloudflare proxy introduces this odd case.

[bdarnell]

After which you get the error Connection" must be "Upgrade".

That doesn't quite make sense - I see Connection: upgrade in those headers, but the upgrade header itself is missing. That should result in the error message "can upgrade only to websocket", instead of "connection must be upgrade".

When you say curl "worked", what exactly do you mean? (when I run those commands now I just get a 301 redirect) curl doesn't support websockets, and I'm not sure it's useful to add enough -H options to curl to implement half of the websocket handshake.

What was the real websocket client that was experiencing these errors? If this is HTTP/2 related, it sounds like it's probably a buggy websocket client that is incorrectly sending the HTTP/2 ALPN token in its TLS handshake (websockets must go over an HTTP/1 connection).

[bdarnell]

(websockets must go over an HTTP/1 connection).

Turns out there's a newish (2 years old) RFC for websockets over HTTP/2: RFC 8441. I don't see anything in the spec that completely lines up with what you're reporting here (websockets over HTTP/2 do away with both the Connection and Upgrade headers), but maybe that's a new angle to explore - could the cloudflare load balancer be terminating the original websocket connection, then trying to forward it in a broken way?

[maartenbreddels]

Hi,

Currently, I don't have the load balancer setup for this server. When I do, I will let you know in this issue so we can explore this again.

Regards,

Maarten

[v3ss0n]

hello @bdarnell any thoughts on merging HTTP2? it is mainstream now and many frameworks (FastAPI/Starlette) have been supporting for long.

[bdarnell]

@v3ss0n That is discussed in #1438, and my comment still stands: a prerequisite to merging HTTP/2 support is someone saying "I've used this and it works well for me".