FORKED: `AsgiWhiteNoise` and async `WhiteNoiseMiddleware`

[Archmonger]

This PR has resulted in a fork: ServeStatic

Tasks

• [x] Create async whitenoise class
• [x] Use aiofiles for asynchronous file handling
• [x] Async Django Middleware
• [x] Compatibility with existing tests
• [x] Add new ASGI-specific tests
• [x] Update the docs to show ASGI configuration

Summary

I've created async variants for any methods/functions/classes that can benefit from aiofiles. For IO operations that currently do not have an async equivalent (such as the OS calls within WhiteNoise.find_file(...)), I've converted to async via asyncio.to_thread(...) (if available).

Django middleware is now sync and async compatible. Thus, the middleware will provide Django an async FileResponse when possible.

WhiteNoise

• whitenoise.base.BaseWhiteNoise
  • Similar to the old WhiteNoise class, but __call__ is now a stub.
• whitenoise.wsgi.WhiteNoise
  • Functionally identical to the old WhiteNoise class. Uses BaseWhiteNoise as a subclass to obtain most methods.
• whitenoise.asgi.AsgiWhiteNoise
  • Implements the same behavior as WhiteNoise, but using the ASGI interface (async/threading)

Responders

• whitenoise.responders.StaticFile
  • Added aget_response (async variant).
  • Added aget_range_response (async variant).
  • Added aget_range_not_satisfiable_response (async variant).
• whitenoise.responders.AsyncSlicedFile
  • Implements the same behavior as SlicedFile, but using the aiofiles context manager interface.
• whitenoise.responders.SlicedFile
  • Minor refactoring to keep visual similarity to AsyncSlicedFile.
• whitenoise.responders.Redirect
  • Added aget_response (async variant).

Middleware

• whitenoise.middleware.WhiteNoiseMiddleware
  • Rewritten to add async support.
• whitenoise.middleware.AsyncWhiteNoiseFileResponse
  • Variant that uses async responses when possible (Django 4.2+ ASGI).
  • Used when Django's middleware system suggests using async
  • Responses must be converted to sync on WSGI.
• whitenoise.middleware.AsyncFileIterator
  • Simple __aiter__ helper that allow Django file responses to use aiofiles.
  • Required to not break WSGI compatibility by providing an isolated async environment for responses.
• whitenoise.middleware.AsyncToSyncIterator
  • Simple __aiter__ to __iter__ converter, since Django cannot use __aiter__ within WSGI.
  • Far more production-ready than the built-in converter on Django 4.2+.

Design Considerations

Code Architecture

• The vast majority of this PR is the addition of new async functions, rather than changing/removal of existing code. This is due to the infectious nature of asyncio.
• We do have the option of allowing ASGI applications directly within WhiteNoise, rather than creating a whole separate AsgiWhiteNoise class. However, this would either rely on heuristics or an initialization arg. This seemed more error-prone than an explicit class.
• Only a minimal subset of methods were converted to async. There is a performance benefit to converting every method in BaseWhiteNoise to async, since event loops perform best with as much yielding via await as possible. However, that's a really high maintenance burden that we shouldn't commit to in this PR.
• I tried to retain as much visual similarity between sync/async variants as possible to improve maintainability. Most async methods match up line-to-line with their sync counterparts. Minor changes were sometimes made to sync code to increase visual similarity.
• Tests were only written to accommodate for the new async-specific bits. Since all non-file methods are still sync, there isn't a particular need to write tests that hit the same code.

Django Integration

• Our middleware will automatically use sync or async responses depending on Django's context-aware sync/async auto-selection. Django tries to reduce the amount of sync/async context switches. If we are preceded by a bunch of sync middleware then Django will tell us to use sync.
• It is necessary for async middleware to be able to convert responses to sync. This is because WSGI supports async middleware but not async file responses. Our conversion is done within a thread, which is far more production-ready than Django core's built-in method of consuming all iterators.
• aiofiles is now a mandatory requirement due to Django middleware being async compatible. The alternative would have been a completely separate middleware class for ASGI, but that would negate a lot of the benefits of Django's middleware's context-aware sync/async auto-selection. Additionally, would have been a significant amount of extra code that would not have been worth the maintenance burden.
• By default, we send ASGI responses containing 8192 byte chunks of file data (same as wsgiref.FileWrapper).

Feature Compatibility

• I did not implement Zero Copy Send support due to no ASGI webservers having compatibility for this yet. No point in adding a feature that can't be used.
• We use asgiref.compatibility.guarantee_single_callable to retain compatibility for both ASGI v2 and ASGI v3.

Test Environment: AsgiWhiteNoise

pip install git+https://github.com/Archmonger/whitenoise@asgi-compat

# example.py
from whitenoise import AsgiWhiteNoise

async def asgi_app(scope, receive, send):
    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [(b'content-type', b'text/plain')],
    })
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })

# Note: This also works with legacy ASGI v2 applications
app = AsgiWhiteNoise(asgi_app, root='static', prefix='static')

# bash
uvicorn example:app

Test Environment: WhiteNoiseMiddleware

pip install git+https://github.com/Archmonger/whitenoise@asgi-compat

The existing middleware has been replaced with an async compatible variant. Therefore, follow the normal documentation for installing WhiteNoiseMiddleware.

Issues

fix #251 fix #179 close #261 related #181

[adamchainz]

Set up pre-commit locally - install as per https://pre-commit.com/ , then pre-commit install in the repo.

[strayer]

Hey, I've been following this PR since I intended to use whitenoise in an async Django project running on uvicorn. I noticed work has been stalled for a while. I'd love to help out to get this complete - is there something I can work on that would help get this forward?

I assume implementing aiofile should be rather straight forward, but I'm not sure what needs to be done to finish the Django ASGI middleware task. Working on documentation probably only makes sense when the final code is done.

[Archmonger]

Hey @strayer

I haven't had as much time to handle this PR as I'd like. The code on this PR is mostly untested. Feel free to fork off my branch and continue development.

[Archmonger]

This PR now has a functional AsyncWhiteNoise class.

Will work on middleware, docs, and tests soon.

[Archmonger]

Django middleware is complete.

Since this repo only supports Django 3.2+, I've converted the existing middleware to be async only.

[Archmonger]

@adamchainz AsgiWhiteNoise has been developed, and I've converted WhiteNoiseMiddleware to be async. I've confirmed via a local test environment that both work as expected.

Before I begin rewriting the Django tests, are you okay with the top-level design behind AsgiWhiteNoise and WhiteNoiseMiddleware?

[Archmonger]

@strayer If you're interested, you can PR my branch with proper tests and documentation.

[robd003]

@adamchainz / @strayer any chance that either of you looked at this yet?

I'd love to get this to a state where it can be merged in for the next release.

[Archmonger]

Did some more testing, looks like there's some funky behavior with the new Django middleware when SecurityMiddleware is above it. I will need to dig into this.

[strayer]

@adamchainz / @strayer any chance that either of you looked at this yet?

I'd love to get this to a state where it can be merged in for the next release.

Hey! Thank you @Archmonger and @robd003 for tagging me! I'm currently pretty busy after switching jobs and going into holiday season, so I won't be able to help test, sorry :(

[Archmonger]

Fixed the issue around SecurityMiddleware. I am going to assume the current implementation is acceptable, unless told otherwise.

I will begin fixing the tests soon.

[Archmonger]

This PR is now compatible with all existing tests.

Adding more tests to cover the new ASGI stuff will come soon.

[Archmonger]

This PR is officially feature complete, fully tested, and fully documented.

Since this PR is fairly bulky, the method to the madness is written within this PR's original post.

[Archmonger]

@evansd Given that I've had to read every file within this repo for the sake of this PR, I wouldn't mind helping out as a maintainer. That is, if you're looking for additional support.

We are planning on integrating AsgiWhiteNoise within ReactPy, so it's in my best interest that this repo stays maintained.

[daniel-brenot]

Bumping just to ask, is there anything left on this PR or can it be merged? It would really help us.

[evansd]

Hi, sorry this has gone unreviewed for so long. Just snowed under with other things at the moment. I am going to try to take a look at this as soon as I can. I think it's an important feature for Whitenoise. But it's a big change and I don't want to just merge it without proper consideration.

[Archmonger]

Since I don't want to get into a cat-and-mouse game of constantly re-merging with main, hopefully this PR can get reviewed soon.

[robd003]

@evansd could you please review this?

[robd003]

Could @adamchainz approve this pull request if @evansd is too busy?

[brianglass]

@Archmonger, I'm running your branch in production in a Django app. I'll let you know if I have any issues, but so far so good.

[robd003]

Can we get this merged in for the 6.7 release @evansd ?

[brianglass]

@Archmonger, when Whitenoise returns a 304 response for a file that is already cached on the client, it gives the following warning:

/Users/bglass/src/orthocal-python/ve/lib/python3.12/site-packages/django/http/response.py:517: Warning: StreamingHttpResponse must consume synchronous iterators in order to serve them asynchronously. Use an asynchronous iterator instead.

Is there something I should be doing to avoid this?

[Archmonger]

What Django version are you running?

[brianglass]

@Archmonger, I'm running Django 5.0b1. I just tried it with 4.2.7 with the same result. I though it might be django-cors-headers since I had the middleware before whitenoise, but I took it out and still got the warning.

[Archmonger]

Can you send me your middleware and installed apps? Also let's pull this discussion to the side so that we don't spam everyone with this conversation.

Discord: totoboto Email: [email protected]

[helenap]

Hey @evansd could you please review and merge this in? This branch has been working just fine for me for months!

[daniel-brenot]

Bumping this since it's been a couple more months. Can we get this merged and released? I'd love to get rid of the errors and sync penalty in my code.

[robd003]

Hey @evansd any update on you looking at this?

[jlost]

Would like to see this!

[duanhongyi]

Can this PR be merged?