gunicorn icon indicating copy to clipboard operation
gunicorn copied to clipboard

Not able to set response timeout in Django app(gunicorn + gevent)

Open alok2k5ingh opened this issue 2 years ago • 4 comments
trafficstars

I have a django app.

Configuration: Cloudfront -> AWS ALB -> Nginx -> Gunicorn(gevent) -> Django App

Versions I'm using: gevent==21.12.0 greenlet==1.1.3.post0 gunicorn==20.1.0

Question: How can I set Timeout for each rest http request so that if the execution for a request takes more than 1 minute than it should return timeout in response and should stop execution of that request in the gevent thread.

Current behaviour: Cloudfront has response timeout set at 60sec so it returns error to client after that but gunicorn(gevent) or django still keeps running precessing the request(sometimes for upto 5 to 6 mins) and hence it's waste of cpu and memory resource. Can even introduce cascade effect due to a bunch of slow requests

Expected Behaviour: gunicorn(gevent) or django kills the gevent request thread(or greenlet) after 60secs and return Timeout error which is returned by ALB and cloudfront.

Test scenario:

For testing, I created these two django views:

def ioTask(request): time.sleep(10.5) return HttpResponse("IO bound task finish!\n")

def cpuTask(request): for i in range(10000000000000): n = iii # return HttpResponse("CPU bound task finish!\n") return HttpResponse("test timeout")

Created this timeout middleware:

from gevent import Timeout from django.http import HttpResponse

class TimeoutMiddleware:
def init(self, get_response):
self.get_response = get_response self.response = None

def __call__(self, request):  
    with Timeout(5, False):  
        self.response = self.get_response(request)
    if self.response is not None:  
        return self.response  
    else:  
        return HttpResponse("Request timed out", status=500)  

Test results:

  1. I observed that timeout is working properly for ioTask view but not for cpuTask view.
  2. cpuTask is running forever
  3. So I'm confused here that how to set timeout for a mixed-type( IO + CPU tasks) or cpu bound tasks?

alok2k5ingh avatar Apr 09 '23 06:04 alok2k5ingh

For clarity, adding screenshots

image image

alok2k5ingh avatar Apr 09 '23 06:04 alok2k5ingh

@benoitc @tilgovi can you please provide some suggestion.

alok2k5ingh avatar Apr 20 '23 03:04 alok2k5ingh

Hi, it will ends after about 1000 seconds, so almost one day... knowing it takes about one second to run 1000000 instructions :)

FLBPeriat avatar Apr 24 '23 04:04 FLBPeriat

@alok2k5ingh: is the following proposal of subclassing GeventWorker a viable solution? https://stackoverflow.com/a/72938801

dbartenstein avatar Oct 09 '23 06:10 dbartenstein