django
Adaptive LIFO mode for the request queue
I'm using Daphne for both http and ws requests for my project, and I've noticed that if my service gets spammed with a surge of requests, they are executed in the order they came in. This is intuitive and fair, however, this effectively shuts my webserver down if the volume of requests is more than it can handle (instead of letting it process as much as it can and fail the rest).
The fundamental problem is this - every request comes with an external timeout (in my case, from the ALB balancer above it), and with a FIFO queue, we're wasting time processing requests that have already timed out OR are close to timing out externally, instead of responding to stuff that the client is still waiting for.
I've seen multiple articles from big tech companies mentioning this, but to my knowledge, they all build custom tools to get this to work (I can't find this setting in daphne / nginx / uwsgi / ALB):
- Using load shedding to avoid overload — AWS Builders' Library
- Building Services at Airbnb, Part 3 — Airbnb Engineering
- Meet Bandaid, the Dropbox service proxy — Dropbox Tech Blog
- Fail at Scale: Reliability in the face of rapid change — Ben Maurer, Facebook
I've also seen this behavior in multiple production projects I maintain.
I built an interactive visualization to help illustrate the point. You can configure the request flow, and then play with FIFO/LIFO/Adaptive LIFO/Max queue size. Check it out here: https://ipeterov.github.io/queue-demo/
https://github.com/user-attachments/assets/02a0902b-cc57-46a6-9295-dbcc6e56fd5d
Do you guys think this makes sense? Or is this a dumb idea?
Hi @ipeterov. Thanks for the question. Interesting.
My first thought is that this would need to be done at the reverse proxy level, rather than in Daphne itself. (So nginx or ALB, but you said you didn't find anything there.)
Beyond that, it's very much a research project. What would it involve? What does a proof of concept look like? (It almost certainly out of scope, but it's interesting, as I said.)
@carltongibson
I'm not super familiar with other tools, here's what I know about ALB and nginx:
- ALB - they used to have a FIFO surge queue in their classic balancers and decided to get rid of them in an explicit design decision - so probably not coming anytime soon, though I messaged the author of the article asking for clarifications
- nginx - they support the queue directive which enables queueing (doesn't happen by default) and even allows to configure max size (available in my simulator, kind of helps compared to unlimited, but worse than LIFO). However, it's a part of their Nginx Plus product which is so enterprisey that I didn't even manage to find a pricing section on the F5 site. Just this random site that says it costs $849/mo (per what?). So even if they do add LIFO, they probably won't move the queue feature to the free tier that everyone uses
The reason I thought about Daphne is that at the moment, as far as I understand, the queuing happens between nginx and Dapne. This is because I'm seeing requests staying in some queue for tens of seconds after they're received by nginx and before they get to my Django code, and I'm pretty sure my ALB and nginx don't have any queueing enabled.
I'll look into it deeper - figure out what exactly is happenning right now (where is the queue located, how does Daphne interact with it), and report back.
So Daphne is built on Twisted. So, likely the answer is lower down there.