gunicorn icon indicating copy to clipboard operation
gunicorn copied to clipboard

Published 20 hours ago • verified publisher icon benoitc

gunicorn not response when there is a lot of CLOSE_WAIT tcp connection

Open ghostHBL opened this issue 3 years ago • 8 comments

I use gunicorn + flask + nginx to deploy my application. Sometime the gunicorn got stuck, but very seldom. There is not any error logs for applications and gunicorn when it got stuck, and, it will get a lot timeout error in http client. I got this problem today, I check the tcp connect status by lsof, I found there is a lot of CLOSE_WAIT connections even reach 1340. I think this problem is cause by the CLOSE_WAIT connections takes up all the tcp resources. How can I resolve this issue? Is it bug of gunicorn?
Sorry about that, I don't know how to reproduct this issue yet, just happen when I try to get some request through http, I just got this issue about 3 time this year. I will try to find the method to reproduct this issue. gunicorn version is 20.0.4

ghostHBL avatar Dec 28 '21 10:12 ghostHBL

Hi I also faced this issue, did you find any solutions?

hafiz031 avatar Jan 25 '22 16:01 hafiz031

Hi I also faced this issue, did you find any solutions?

Nope. And, still can not reproduct this issue.

ghostHBL avatar Jan 26 '22 02:01 ghostHBL

I had a similar issue. Gunicorn uses sync workers by default. I switched to gthread and experimented with my worker count until the issue went away.

UPDATE: After some more reading, I learned that the async (gthread) workers deliberately leave the TCP connection open, to avoid having to renegotiate new connections. Since this is not what I need, I switched back to sync and found the right number of workers to handle my loads. Turns out that a slow network or client can really effect service. To get around these situations, output from Gunicorn should be buffered by a reverse proxy that sits in front of it.

glroman avatar May 05 '22 20:05 glroman

Face the same issue

eviltomato avatar Aug 18 '22 06:08 eviltomato

Since a week now we're also facing same issue.

Our stack is deployed on AWS Elastic beanstalk with django+gunicorn+nginx

We tried switching to Apache httpd, but after few days encountered the same issue. Lot of connections are in close_wait state.

amitchavan90 avatar Sep 30 '23 08:09 amitchavan90

Can you share the configuration and version of nginx?

benoitc avatar Oct 01 '23 07:10 benoitc

Face the same issue, and the bug can easily reproduce with the following settings:

  1. set threads(like 2) <= worker_connections(like 3)
  2. just set worker num=1, and worker_type=gthread
  3. continuously launch 2 clients to send request to the server. Up to now, everything OK
  4. than launch the 3rt client, the server hangs there. ctrl+c the 3rt client, then we can see CLOSE_WAIT in the output of "lsof -i"

wykvictor avatar Mar 29 '24 10:03 wykvictor