aiohttp doesn't comply autobahn test suite

[asvetlov]

Thanks @Nothing4You for raising the question and @njsmith for the fix suggestion.

are autobahn tests not used anymore?

Richard Schwab @Nothing4You Oct 20 21:30
also, is ws_connect only supported using a ClientSession?

Andrew Svetlov @asvetlov 01:26
autobahn workflow required manual steps. If you want to integrate them into aiohttp test suite -- you are welcome!
is ws_connect only supported using a ClientSession

Sorry, I don't follow. What did you expect? ClientSession is the root for client-side API


Richard Schwab @Nothing4You 01:52
forgot that a session is also required for other stuff
thought it was just for convenience
i just noticed that the autobahn tests have code that doesn't work anymore

Nathaniel J. Smith @njsmith 06:19
running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance

At least we should run autobahn manually and fix it; ideally automate it.

[webknjaz]

running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance

@anesabml I think this could be a starting point for automating the test runs in our CIs :point_up:

[anesabml]

@webknjaz it's a great example, and it got me thinking if we could just use docker-py / python-on-whales instead of pytest-docker-compose to start the docker containers. (I am sorry if this is taking too long but I just want to make sure that I am doing the right thing)

[webknjaz]

Yeah, maybe. Honestly, I'd be happy to avoid a vendor lock and make sure that it also works with a rootless podman too. But if the libs don't support it, hitting up docker directly is fine by me.

[Dreamsorcerer]

What's still needed to get these tests passing? It's rather a shame that >10% of our CI time is spent running xfail tests. Maybe it's better to disable these tests for now? Or, only run them in specific scenarios (like push to master, so it doesn't run on PRs)?

[adi928]

I started tracking down what is needed to fix the tests. Apparently, the tests were never failing. Working with a docker setup, the tests do pass. So the next thing to do is to follow up on this:

running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance

Following this PR in wsproto repository, they seem to have lost faith in it. Primarily because autobahn only supports till Python2. I think that's the case here too.

Steps used:

1. docker build -f tests/autobahn/Dockerfile.autobahn -t autobahn-testsuite .
2. python3 -m pip install -e .
3. python3 tests/autobahn/server/server.py Then fina step:

docker run --rm \
  -v "$(pwd)/tests/autobahn/server:/config" \
  -v "$(pwd)/autobahn-reports:/reports" \
  autobahn-testsuite \
  wstest --mode fuzzingclient --spec /config/fuzzingclient_case12.json

fuzzingclient_case12.json:

{
    "options": { "failByDrop": false },
    "outdir": "./reports/servers",

    "servers": [
        {
            "agent": "AutobahnServer",
            "url": "ws://host.docker.internal:9001",
            "options": { "version": 18 }
        }
    ],

    "cases": ["12.*"],
    "exclude-cases": [],
    "exclude-agent-cases": {}
}

Porting docker to python-on-whales can be a different issue/task. Tagging @webknjaz since you had activity on this issue last.

[webknjaz]

@adi928 thanks for looking into that. Unfortunately, I can't participate right now. Not for weeks, likely — I'm mostly away from GitHub currently.

But if you come up with some demo PRs, I'm sure @bdraco and @Dreamsorcerer will be able to give feedback and strategize. If you'd like more interactive comms, stop by the matrix space/channels.