actix-web
actix-web copied to clipboard
autobahn websocket test suite tracking
To ensure that actix-web provides production-grade websockets, it should pass the crossbar autobahn test suite.
https://github.com/crossbario/autobahn-testsuite
Currently failing cases:
- [ ] 3.1 - Issue #1579, PR #xx
- [ ] 3.2
- [ ] 3.3
- [ ] 3.4
- [ ] 3.5
- [ ] 3.6
- [ ] 5.3
- [ ] 5.4
- [ ] 5.5
- [ ] 5.6
- [ ] 5.7
- [ ] 5.8
- [ ] 5.18
- [ ] 5.19
- [ ] 5.20
- [ ] 6.1.2
- [ ] 6.1.3
- [ ] 6.2.2
- [ ] 6.2.3
- [ ] 6.2.4
- [ ] 6.3.2
- [ ] 6.4.1
- [ ] 6.4.2
- [ ] 6.4.3
- [ ] 6.4.4
- [x] 7.1.3 - user dependent: closed in actix/examples#335
- [x] 7.1.4 - user dependent: closed in actix/examples#335
- [ ] 7.9.1
- [ ] 7.9.2
- [ ] 7.9.3
- [ ] 7.9.4
- [ ] 7.9.5
- [ ] 7.9.6
- [ ] 7.9.7
- [ ] 7.9.8
- [ ] 7.9.9
- [ ] 7.9.10
- [ ] 7.9.11
- [ ] 9.1.2
- [ ] 9.1.3
- [ ] 9.1.4
- [ ] 9.1.5
- [ ] 9.1.6
- [ ] 9.2.2
- [ ] 9.2.3
- [ ] 9.2.4
- [ ] 9.2.5
- [ ] 9.2.6
- [ ] 9.3.1
- [ ] 9.3.2
- [ ] 9.3.3
- [ ] 9.3.4
- [ ] 9.3.5
- [ ] 9.3.6
- [ ] 9.3.7
- [ ] 9.3.8
- [ ] 9.3.9
- [ ] 9.4.1
- [ ] 9.4.2
- [ ] 9.4.3
- [ ] 9.4.4
- [ ] 9.4.5
- [ ] 9.4.6
- [ ] 9.4.7
- [ ] 9.4.8
- [ ] 9.4.9
- [ ] 9.5.1
- [ ] 9.5.2
- [ ] 9.5.3
- [ ] 9.5.4
- [ ] 9.5.5
- [ ] 9.5.6
- [ ] 9.6.1
- [ ] 9.6.2
- [ ] 9.6.3
- [ ] 9.6.4
- [ ] 9.6.5
- [ ] 9.6.6
- [ ] 10.1.1
I'd be happy to take this on if no one else has. I imagine this starts by making some sort of reasonable ws example. I currently have this which is near-randomness as I don't understand the domain very well yet.
Then I ran the autobahn test suite and got these results (Edit: rendered results). Unsurprisingly there are lots of failures. The fixes for those failures are going to presumably go into different repos (like actix
or actix-web
or what-have-you) but will be tested in examples
. I'm not exactly sure how to set up Cargo
to make that work.
So I guess what I might need help with is:
- Setting up a good ws example to test against
- Learning how to update
actix
/actix-web
and use the updated version of those packages to test the example (or maybe the example needs to be moved?)
Once I have that all set up I'd be happy to create a checklist of all the test failures that need to be fixed and start knocking them off/opening the floor for others to pick them off.
Just bumping this again to see if anyone has time helping me with those two steps above so I can get kicking on fixing these issues!
I made https://github.com/actix/examples/pull/331 as hopefully the place where we can start
Setting up a good ws example to test against
Once that's set, I'll document the current failures in this issue and we can figure out how to work on fixing them!
[list moved to issue description by admin]
The changes made in PR actix/examples#335 have caused many improvements to close behavior but have highlighted some issues wrt invalid close code handling. When that PR is merged we may check off cases: 7.1.3 & 7.1.4 but will need to add the entirety of section 7.9.
Screenshots of test results.
Tests with -fixed-1 suffix, to the right, show the new behavior, with close codes sent.
Note that 7.1.3 and 7.1.4 are shown failing but will be passing before the PR is merged.
![2020-06-24_01 07 06-8f92b67a](https://user-images.githubusercontent.com/3316789/85480509-0f170c00-b5b8-11ea-8eca-88c9a240e905.png)
![2020-06-24_01 07 23-6fcf6539](https://user-images.githubusercontent.com/3316789/85480519-1211fc80-b5b8-11ea-9920-c728ef3388fe.png)
Hi, I'm new to the actix-web, so sorry for silly questions.
I've created a repo https://github.com/mskorkowski/actix-web-websocket-echo where you can find a code with echo server handling all except 7 test cases below:
- Test cases 3.* require access to the reserved bits so they are unfixable on the echo server side.
- 7.5.1 failure would require changes in the
CloseReason
. CurrentlyCodec
is performig lossy conversion to utf8, which makes it impossible to detect if the description part of theclose
frame is valid or not.
My question are
- Are you interested in merging this code with the autobahn echo example? If so what is the proper procedure to follow?
- Should I open a ticket for 7.5.1 for detailed discussion?