autobahn websocket test suite tracking

[rustrust]

To ensure that actix-web provides production-grade websockets, it should pass the crossbar autobahn test suite.

https://github.com/crossbario/autobahn-testsuite

Currently failing cases:

• [ ] 3.1 - Issue #1579, PR #xx
• [ ] 3.2
• [ ] 3.3
• [ ] 3.4
• [ ] 3.5
• [ ] 3.6
• [ ] 5.3
• [ ] 5.4
• [ ] 5.5
• [ ] 5.6
• [ ] 5.7
• [ ] 5.8
• [ ] 5.18
• [ ] 5.19
• [ ] 5.20
• [ ] 6.1.2
• [ ] 6.1.3
• [ ] 6.2.2
• [ ] 6.2.3
• [ ] 6.2.4
• [ ] 6.3.2
• [ ] 6.4.1
• [ ] 6.4.2
• [ ] 6.4.3
• [ ] 6.4.4
• [x] 7.1.3 - user dependent: closed in actix/examples#335
• [x] 7.1.4 - user dependent: closed in actix/examples#335
• [ ] 7.9.1
• [ ] 7.9.2
• [ ] 7.9.3
• [ ] 7.9.4
• [ ] 7.9.5
• [ ] 7.9.6
• [ ] 7.9.7
• [ ] 7.9.8
• [ ] 7.9.9
• [ ] 7.9.10
• [ ] 7.9.11
• [ ] 9.1.2
• [ ] 9.1.3
• [ ] 9.1.4
• [ ] 9.1.5
• [ ] 9.1.6
• [ ] 9.2.2
• [ ] 9.2.3
• [ ] 9.2.4
• [ ] 9.2.5
• [ ] 9.2.6
• [ ] 9.3.1
• [ ] 9.3.2
• [ ] 9.3.3
• [ ] 9.3.4
• [ ] 9.3.5
• [ ] 9.3.6
• [ ] 9.3.7
• [ ] 9.3.8
• [ ] 9.3.9
• [ ] 9.4.1
• [ ] 9.4.2
• [ ] 9.4.3
• [ ] 9.4.4
• [ ] 9.4.5
• [ ] 9.4.6
• [ ] 9.4.7
• [ ] 9.4.8
• [ ] 9.4.9
• [ ] 9.5.1
• [ ] 9.5.2
• [ ] 9.5.3
• [ ] 9.5.4
• [ ] 9.5.5
• [ ] 9.5.6
• [ ] 9.6.1
• [ ] 9.6.2
• [ ] 9.6.3
• [ ] 9.6.4
• [ ] 9.6.5
• [ ] 9.6.6
• [ ] 10.1.1

[mlodato517]

I'd be happy to take this on if no one else has. I imagine this starts by making some sort of reasonable ws example. I currently have this which is near-randomness as I don't understand the domain very well yet.

Then I ran the autobahn test suite and got these results (Edit: rendered results). Unsurprisingly there are lots of failures. The fixes for those failures are going to presumably go into different repos (like actix or actix-web or what-have-you) but will be tested in examples. I'm not exactly sure how to set up Cargo to make that work.

So I guess what I might need help with is:

1. Setting up a good ws example to test against
2. Learning how to update actix / actix-web and use the updated version of those packages to test the example (or maybe the example needs to be moved?)

Once I have that all set up I'd be happy to create a checklist of all the test failures that need to be fixed and start knocking them off/opening the floor for others to pick them off.

[mlodato517]

Just bumping this again to see if anyone has time helping me with those two steps above so I can get kicking on fixing these issues!

[mlodato517]

I made https://github.com/actix/examples/pull/331 as hopefully the place where we can start

Setting up a good ws example to test against

Once that's set, I'll document the current failures in this issue and we can figure out how to work on fixing them!

[mlodato517]

[list moved to issue description by admin]

[robjtede]

The changes made in PR actix/examples#335 have caused many improvements to close behavior but have highlighted some issues wrt invalid close code handling. When that PR is merged we may check off cases: 7.1.3 & 7.1.4 but will need to add the entirety of section 7.9.

Screenshots of test results.

Tests with -fixed-1 suffix, to the right, show the new behavior, with close codes sent.
Note that 7.1.3 and 7.1.4 are shown failing but will be passing before the PR is merged.

[mskorkowski]

Hi, I'm new to the actix-web, so sorry for silly questions.

I've created a repo https://github.com/mskorkowski/actix-web-websocket-echo where you can find a code with echo server handling all except 7 test cases below:

• Test cases 3.* require access to the reserved bits so they are unfixable on the echo server side.
• 7.5.1 failure would require changes in the CloseReason. Currently Codec is performig lossy conversion to utf8, which makes it impossible to detect if the description part of the close frame is valid or not.

My question are

1. Are you interested in merging this code with the autobahn echo example? If so what is the proper procedure to follow?
2. Should I open a ticket for 7.5.1 for detailed discussion?