Suggestion: ASGI compliance tests

[simonw]

I'm always excited to see new ASGI hosting implementations emerge - today I've been trying out this wrapper for Azure Functions for example: https://github.com/Azure/azure-functions-python-library/issues/75

It would be great if there was a really robust way to put these new implementations through their paces. asgiref seems like it might be the right place for that to exist.

My initial idea is a solution with two parts:

• An ASGI application function that takes the incoming scope and receive arguments, reads from them and sends back a big JSON response derived from what came in
• A Python script that can be pointed at that hosted function and then sends a carefully curated set of requests to it and verifies that the response comes back as expected

I have a ASGI function that returns the repr of the scope which I use to test out new implementations by eyeballing them - eg here: https://azure-functions-datasette.azurewebsites.net/-/asgi-scope - I'm imagining something a bit like that which pairs up with a asgi-test https://azure-functions-datasette.azurewebsites.net/-/asgi-scope command-line script (which calls a Python function).

[simonw]

So three questions:

• Does this already exist somewhere?
• If not, should this live in asgiref or in a separate project?
• Anyone interested in working with me to build this thing?

[simonw]

I think there's a lot to be said for implementation-independent test suites like http://acid3.acidtests.org/

[simonw]

It strikes me that the first version of this doesn't need to be 100% comprehensive to be useful - just covering things like the default scope keys and making sure they are bytes or strings would be a big win for people trying to spin up new ASGI implementations.

I'd love to build asgi-to-wsgi (see #109) and this would make it a lot easier!

[andrewgodwin]

There isn't one of these, and I do agree there should be - I've usually called these "acceptance" or "compliance" test suites, and it would help a lot in enforcing that the spec is actually adhered to.

It'd be a little tricky to test both directions given the async nature of both ASGI and its servers - making it robust in the face of all the different event loop implementations is tricky - but it's probably possible.

[simonw]

I like the term "compliance" for this. Getting it to work for all aspects of async ASGI does sound tricky, but I think just the basics of the HTTP spec (which is the bit that I'm seeing the most activity around) should be achievable without too much trouble.

[Kludex]

Are you still interested in this @simonw? I'm willing to help here.