Websockets 1002 Unknown opcode: 7 with Jetty 10.0.24 on Mac, but not Linux, Win10 - with test case attached

[afarber]

trafficstars

Jetty version(s) 12.0.14

Java version/vendor (use: java -version) javac 23

OS type/version macOS Sequoia

How to reproduce?

Please run the ./test-mac.sh script in the attached zip archive and then open the URL http://127.0.0.1:8080/en/ in Edge or Firefox browser on macOS and look at the browser console:

jetty-mac-opcode-7.zip

The test Javascript code is in the file jetty-mac-opcode-7/my-servlet/src/main/resources/index.html

Thank you

[afarber]

Also, I tested downgrading to Jetty 10.0.22 and 10.0.20, but the above test case fails there too.

[sbordet]

@afarber sorry, but Jetty 10/11 are at end of community support:

• #10485

Please upgrade to Jetty 12 and report back if you still have the problem.

[afarber]

Yes, this bug seems to be in Jetty 12 too:

2024-10-18 22:19:54.608:WARN :oe.WebSocketChatServlet$ChatWebSocket:qtp369241501-43: Unhandled Error: Endpoint org.example.WebSocketChatServlet$ChatWebSocket : org.eclipse.jetty.websocket.api.exceptions.ProtocolException: Unknown opcode: 7

To reproduce:

mkdir /Users/afarber/src/jetty-questions/test-demo/jetty.base
cd /Users/afarber/src/jetty-questions/test-demo/jetty.base
java -jar /Users/afarber/src/jetty-questions/jetty-home-12.0.14/start.jar --add-modules=server,http,ee10-deploy,ee10-demos
java -jar /Users/afarber/src/jetty-questions/jetty-home-12.0.14/start.jar

And then just navigate to the URLs:

• http://127.0.0.1:8080/ee10-test/jetty.websocket/jakarta.websocket/ or
• http://127.0.0.1:8080/ee10-test/jetty.websocket/jetty.websocket/

I have tried both Edge and Safari browsers on macOS 15

[afarber]

[joakime]

@afarber can you report the HTTP Upgrade Request and Response headers from your developer tools on safari and edge? We want to see the details on how the browser attempts to negotiate the websocket endpoint.

[joakime]

Noting the websocket bugs listed for webkit (used by both edge and safari on macos)

• https://bugs.webkit.org/show_bug.cgi?id=202401
• https://bugs.webkit.org/show_bug.cgi?id=228296
• https://bugs.webkit.org/show_bug.cgi?id=230962

Also, what is the exact versions listed for your safari and edge browsers?

[afarber]

Microsoft Edge Version 129.0.2792.89 (Official build) (arm64)

Safari Version 18.0.1 (20619.1.26.31.7)

Firefox 131.0.3 (aarch64)

[afarber]

Edge request headers:

GET ws://127.0.0.1:8080/ee10-test/jetty.websocket/jetty.websocket/ HTTP/1.1
Host: 127.0.0.1:8080
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36 Edg/129.0.0.0
Upgrade: websocket
Origin: http://127.0.0.1:8080
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,de;q=0.7,ru;q=0.6,de-DE;q=0.5
Cookie: visited=yes
Sec-WebSocket-Key: N5B6trQcdmryzryWUqdZyQ==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: chat

Edge response headers:

HTTP/1.1 101 Switching Protocols
Server: Jetty(12.0.14)
Vary: Accept-Encoding
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: IREMkPTB9YEubw1Inc5u5EcngRY=

[afarber]

Safari request headers:

Cache-Control: no-cache
Connection: Upgrade
Origin: http://127.0.0.1:8080
Pragma: no-cache
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: zzKJ9wqRdw9v59uE3NyWKQ==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15

Safari response headers:

Connection: Upgrade
Sec-WebSocket-Accept: h+N0SkVKm0GK9waNAkYliyZCIaE=
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Protocol: chat
Server: Jetty(12.0.14)
Upgrade: websocket
Vary: Accept-Encoding

[joakime]

Yup, you have websocket compression via the permessage-deflate extension in both cases. Something the webkit bugs report as an issue in both cases.

If you use something like Chrome on that same Mac, does it also fail? (it has a different underlying web technology codebase).

Oddly, the reported browser, in both cases is "Macintosh; Intel Mac OS X 10_15_7", which isn't a M1 mac. But that's just a strange thing I noticed, doesn't really matter, or mean anything.

[afarber]

Firefox request headers:

GET /ee10-test/jetty.websocket/jetty.websocket/ HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Prefer: safe
Sec-WebSocket-Version: 13
Origin: http://127.0.0.1:8080
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: HGAIInfKTFMKRKdqe0jY+g==
Connection: keep-alive, Upgrade
Cookie: visited=yes
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Firefox response headers:

HTTP/1.1 101 Switching Protocols
Server: Jetty(12.0.14)
Vary: Accept-Encoding
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: mdsEAZ3aa4Mb2Yv5DDFP15eBRxs=

[afarber]

Sorry, I cannot install Chrome, because I do not want them to track my Android dev account or ban Google/Android phone accounts of me and my relatives.

They already banned my AdSense account for nothing maybe 15 years ago and the experience was bad.

[joakime]

I only have a Macbook Pro from 2012, can't upgrade it to Seqouia.

Seeing as websocket compression is just continually hosed on Safari and webkit, it's probably time to implement a User-Agent based compression logic at the low level websocket negotiation code.

We are not the only group struggling with webkit issues with websocket compression.

• https://github.com/gobwas/ws/issues/169
• https://github.com/gorilla/websocket/issues/731
• https://github.com/coder/websocket/issues/218

The three User-Agents look like ...

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36 Edg/129.0.0.0

I think we should look for either Mac OS X or AppleWebKit in the User-Agent as a trigger to disable compression negotiation.

@lachlan-roberts what do you think about having a configurable for something like "websocket.compression.webkit.enabled=false" as a default.

[joakime]

Ya know, maybe a WebkitDisableWebsocketCompressionRule makes sense too. It would strip out the Sec-WebSocket-Extensions: permessage-deflate from the client side.

Similar to what we used to have back in Jetty 9, with the MsieSslRule.

It could be enabled rather quickly from a jetty-home module too.

[afarber]

Wouldn't AppleWebKit also match iOS Safari?

I have tried with my iPhone 13 and the demo seems to work there:

[lachlan-roberts]

I cannot reproduce the issue at all on my Mac. But it is a 2015 model so won't let me update to latest OS, also my machine is Intel Mac not M1.

I can join the websocket chatroom and send messages with no errors on Safari, Firefox and Edge.

Here are my results for comparison.

System Specs

macOS Monterey version 12.7.6
MacBook Pro (Retina, 13-inch, Early 2015)
Processor: 3.1 GHz Dual-Core Intel Core i7
Memory: 16 GB 1867 MHz DDR3

Firefox 131.0.3 (64-bit)

Request Headers

GET /ee10-test/jetty.websocket/jetty.websocket/ HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Sec-WebSocket-Version: 13
Origin: http://127.0.0.1:8080
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: GTxkxfdUSBXURCyQeVhaEw==
Connection: keep-alive, Upgrade
Cookie: visited=yes
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Response Headers

HTTP/1.1 101 Switching Protocols
Server: Jetty(12.0.15-SNAPSHOT)
Vary: Accept-Encoding
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Upgrade: websocket
Sec-WebSocket-Accept: Zq4jmDxG2KCTfMPZvSX4eSRDhW8=
Connection: Upgrade

Edge Version 130.0.2849.46 (Official build) (x86_64)

Request Headers

GET ws://127.0.0.1:8080/ee10-test/jetty.websocket/jetty.websocket/ HTTP/1.1
Host: 127.0.0.1:8080
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0
Upgrade: websocket
Origin: http://127.0.0.1:8080
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-AU,en-GB;q=0.9,en;q=0.8,en-US;q=0.7
Cookie: visited=yes
Sec-WebSocket-Key: Gpy1pqEDJZ77znzH4tmjWA==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: chat

Response Headers

HTTP/1.1 101 Switching Protocols
Server: Jetty(12.0.15-SNAPSHOT)
Vary: Accept-Encoding
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Extensions: permessage-deflate
Upgrade: websocket
Sec-WebSocket-Accept: QMOUYHLnudNllpnggTh7Od80Oww=
Connection: Upgrade

Safari Version 17.6 (17618.3.11.11.7, 17618)

Request Headers

Cache-Control: no-cache
Connection: Upgrade
Origin: http://127.0.0.1:8080
Pragma: no-cache
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: dOEYP6yveZkdyW9hWFhGHA==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15

Response Headers

Connection: Upgrade
Sec-WebSocket-Accept: fwjTcXX69EOzx9rciBGGjF1obnU=
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Protocol: chat
Server: Jetty(12.0.15-SNAPSHOT)
Upgrade: websocket
Vary: Accept-Encoding

[lachlan-roberts]

I think we should look for either Mac OS X or AppleWebKit in the User-Agent as a trigger to disable compression negotiation.

@joakime I'm not sure we should just disable compression for all Mac OS. Unless we find know why its actually failing and that is not a bug in Jetty.

My User-Agent headers were exactly the same and there was no failure.

[lachlan-roberts]

@afarber as a workaround you could just disable websocket compression in your JettyWebSocketCreator.

Try clearing out the extensions on the JettyServerUpgradeResponse with resp.setExtensions(Collections.emptyList()).

[afarber]

@lachlan-roberts I have tried your suggestion to filter out "permessage-deflate" and it is not helping in my test project:

Edge request headers:

GET ws://localhost:8080/sandbox HTTP/1.1
Host: localhost:8080
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0
Upgrade: websocket
Origin: http://localhost:8080
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,de;q=0.7,ru;q=0.6,de-DE;q=0.5
Sec-WebSocket-Key: bej660HPl+egTzwQlrvkqg==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

Edge response headers (without "permessage-deflate"):

HTTP/1.1 101 Switching Protocols
Server: Jetty(12.0.14)
Date: Tue, 22 Oct 2024 09:05:40 GMT
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: UTH9q9PiLHiGTWeS+IKVSwRwl/Y=

Jetty logs:

WebSocket server is running - ws://localhost:8080/sandbox
Open in browser to test - http://localhost:8080
2024-10-22 11:05:40.322:INFO :SandboxWebSocketServlet:qtp1166172176-42: User-Agent: true [permessage-deflate;client_max_window_bits] []
2024-10-22 11:05:40.368:INFO :SandboxWebSocket:qtp1166172176-42: CONNECTED: /[0:0:0:0:0:0:0:1]:63854
2024-10-22 11:05:43.210:INFO :SandboxWebSocket:qtp1166172176-41: ERROR: {}
org.eclipse.jetty.websocket.api.exceptions.ProtocolException: Unknown opcode: 7

[joakime]

@joakime I'm not sure we should just disable compression for all Mac OS. Unless we find know why its actually failing and that is not a bug in Jetty.

My User-Agent headers were exactly the same and there was no failure.

If you look around other issue trackers (outside of Jetty), you will quickly see that the support for websocket compression on Mac OS has been more often broken across its history than it is working. Disabling websocket compression for all Mac OS browsers as a blanket configuration is actually quite common. (simply because its far too complicated to narrow down specific combinations of browser + browser versions + OS architecture + OS version where the issue exists, vs where it works)

[lachlan-roberts]

@afarber can you please collect full Jetty debug logs for when the issue occurs and post it here.

[joakime]

@afarber you can run the chat demo on ee10-demos with the command line ...

java -Dorg.eclipse.jetty.LEVEL=DEBUG -jar /path/to/jetty-home/start.jar

to have it output in DEBUG mode.

Or you can do the following, which might be easier.

$ mkdir demos-12-ee10
$ cd demos-12-ee10
$ java -jar ~/code/jetty/distros/jetty-home-12.0.12/start.jar --add-module=http,ee10-demos,console-capture
$ chmod u+w resources/jetty-logging.properties
$ echo "org.eclipse.jetty.LEVEL=DEBUG" > resources/jetty-logging.properties
$ java -jar ~/code/jetty/distros/jetty-home-12.0.12/start.jar

Run the tests in your browser, and then Ctrl+C the running Jetty. Check the logs/ directory for the raw DEBUG logs. Compress the logs (eg: gzip -9 <filename> or zstd -15 <filename>) and attach them here please.

[afarber]

Here you go @joakime , do you want wireshark logs captured too?

2024_10_22.jetty.log.gz

[joakime]

Here you go @joakime , do you want wireshark logs captured too?

Thanks! Yeah, the wireshark might help us understand the failure more. Can you attach that too?

[afarber]

The output of sudo dumpcap -i lo0 -w wireshark.pcap:

wireshark.pcap.gz

The output of tshark -r wireshark.pcap -V > wireshark.txt:

wireshark.txt

2024_10_22.jetty.log.gz

[lachlan-roberts]

It looks like Jetty is receiving the upgrade request twice in a row on the same connection, and trying to parse it as WebSocket but it is really just HTTP/1.1 bytes.

There seems to be some intermediary on port 10011 which is first receiving the weboscket upgrade request and then forwarding it on to jetty at port 8080. And this is what sends the double HTTP GET request to Jetty which is incorrect as it will be interpreted as websocket data.

[afarber]

Yes, I have something there:

% netstat -an | grep 10011 | grep LISTEN
tcp4       0      0  *.10011                *.*                    LISTEN     
tcp6       0      0  *.10011                *.*                    LISTEN  

It is Microsoft Edge and also Microsoft Teams

% lsof +c 20 -i :10011

COMMAND               PID    USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
Microsoft\x20Edge    3419 afarber   20u  IPv4 0x93f2ab4e4332e0f6      0t0  TCP localhost:64624->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   21u  IPv4 0x3846e7ab2ea7f734      0t0  TCP localhost:64669->localhost:10011 (CLOSE_WAIT)
Microsoft\x20Edge    3419 afarber   24u  IPv6 0xcb07d7fc7fb8c6c8      0t0  TCP localhost:64639->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   26u  IPv6 0x5e65bdae7bb8dd3c      0t0  TCP localhost:64630->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   30u  IPv4 0x299de8efbc43efd2      0t0  TCP localhost:64665->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   34u  IPv4 0xc663c28a01cfa2d8      0t0  TCP localhost:64649->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   36u  IPv6 0xa2472c4370661bc2      0t0  TCP localhost:64645->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   37u  IPv4 0x3f9cde4d9d4addc6      0t0  TCP localhost:64653->localhost:10011 (ESTABLISHED)
Microsoft\x20Edge    3419 afarber   49u  IPv4 0x5f4d3dc43e5b88a8      0t0  TCP localhost:64663->localhost:10011 (ESTABLISHED)
MSTeams              3600 afarber   53u  IPv4  0x92433cab197a4b0      0t0  TCP localhost:64697->localhost:10011 (ESTABLISHED)
MSTeams              3600 afarber   54u  IPv4 0x668d956acc72c044      0t0  TCP localhost:64717->localhost:10011 (ESTABLISHED)
MSTeams              3600 afarber   56u  IPv6 0x88fa37c0c4868f00      0t0  TCP localhost:64706->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   22u  IPv6 0x545c8562943994db      0t0  TCP localhost:64671->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   25u  IPv6 0x7c80f69291d3903c      0t0  TCP localhost:64778->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   26u  IPv4 0x23ef410497e6f02f      0t0  TCP localhost:64802->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   28u  IPv4 0x6485bae5d44652d3      0t0  TCP localhost:64677->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   29u  IPv6 0x39e51bd897932ebe      0t0  TCP localhost:64781->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   30u  IPv6 0x1ce7a8d5c5bd39ee      0t0  TCP localhost:64804->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   34u  IPv4 0x9875781c983adfea      0t0  TCP localhost:64790->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   35u  IPv6 0x252a8fc5a7f2d579      0t0  TCP localhost:64795->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3626 afarber   50u  IPv4 0xa969791744e230f5      0t0  TCP localhost:64719->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3640 afarber   49u  IPv4  0x5142b498168ebf9      0t0  TCP localhost:64733->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3640 afarber   50u  IPv4 0x96325b3a2a31b610      0t0  TCP localhost:64735->localhost:10011 (ESTABLISHED)
Microsoft\x20Teams   3640 afarber   51u  IPv6 0x6a96f10f1cf08fb9      0t0  TCP localhost:64739->localhost:10011 (ESTABLISHED)

Only when I quite both programs using CMD-Q, then no more lines printed by the lsof command.

And when I start them, they use the same port again.

Also I have this:

[lachlan-roberts]

I think privado networks llc is a VPN, so this is potentially causing the issue.

[lachlan-roberts]

Either way it doesn't seem to be a bug in Jetty.

[afarber]

Yes, Privado VPN client, is not active though... So maybe it is Microsoft Teams, Edge or Microsoft AutoUpdater...

Sure it is not a Jetty bug and sorry for reporting it as one initially.

Wouldn't it be nice though to have an optional Jetty setting to ignore duplicated HTTP upgrade requests?