File Descriptors are left open as deleted

[AvnerCohen]

So, surely this is an ongoing issue for years, happy to provide any additional info.

I guess this is related to: https://github.com/benoitc/gunicorn/pull/1428 https://github.com/benoitc/gunicorn/pull/1310 And specifically: https://github.com/benoitc/gunicorn/issues/1327

What I see with LSOF:

gunicorn: 28994    root   85u   REG     202,1      4096   4410255 /tmp/ffi0UYHeR (deleted)
gunicorn: 28994    root   87u   REG     202,1      4096   4418003 /tmp/fficKN0nq (deleted)
gunicorn: 28994    root   89u   REG     202,1      4096   4410256 /tmp/ffiCJmoAL (deleted)
gunicorn: 28994    root   91u   REG     202,1      4096  12512660 /tmp/ffieVk8pD (deleted)
gunicorn: 28994    root   93u   REG     202,1      4096   4423883 /tmp/ffizqkh3G (deleted)
gunicorn: 28994    root   95u   REG     202,1      4096   8838233 /tmp/ffiQwK7gn (deleted)
gunicorn: 28994    root   97u   REG     202,1      4096  12518372 /tmp/ffidaY02N (deleted)
gunicorn: 28994    root   99u   REG     202,1      4096   4425587 /tmp/ffilhE4JC (deleted)
gunicorn: 28994    root  101u   REG     202,1      4096  12518403 /tmp/ffijQimeC (deleted)
gunicorn: 28994    root  103u   REG     202,1      4096  12518371 /tmp/ffiPkL3qy (deleted)
gunicorn: 28994    root  105u   REG     202,1      4096   8829268 /tmp/ffiixrJ49 (deleted)

Many hundreds of these.

I run gunicorn 19.8.0 on python 2.7.14 and CentOs7

I serve standard HTTP requests from a Pyramid app.

[tilgovi]

@AvnerCohen thank you for reporting this and for taking the time to link to other issues.

Is 28994 the PID there of the arbiter?

[AvnerCohen]

Here is another dump of this:

$ lsof -c gunicorn | egrep "deleted" | awk {'print $2 '}  | sort | uniq -c | sort -n
     16 29766
     16 29767
     16 29768
     16 29771
     16 29773
     16 29774
     16 29776
     16 29777
     16 29778
     24 29761

This is after a restart (using USR2>WINCH>QUIT). PID = 29761 is the master all others are the workers attached to it. So it looks like all elements are leaking. What I did notice is that on each "restart" increases the leaked FDs, so maybe it's something around that.

lmk if anything else I can test or provide. Thanks for the quick response!

[alochym01]

i got same error, is there any patches to fix

thank you

[alochym01]

@AvnerCohen do you have a work around for this issue

[AvnerCohen]

No. No real workaround, just better monitoring to fully stop/strt every now and then.

[benoitc]

what happen if you'using latest 19.x or 20.x (master) ?

[benoitc]

bump

[AvnerCohen]

@benoitc As you can see in the first post, we already use 19.x (19.8.0). However, we have recently upgraded to python 3.x and this does no longer seems to be an issue.

So I am guessing , since gunicorn is dropping python 2.x support, this basically can be closed.

Thanks for checking and following up !

[shantha-Ekanayake]

Even with the python 3.x I am still facing issues of having temp files in lsof and holding resources. Did anyone find any solution to this problem?

[javabrett]

@shantha-Ekanayake this is an old issue that should probably have been closed - can you provide your versions for everything, esp. Python and Gunicorn, and some recorded observations and logs?

[Zoom2016]

@shantha-Ekanayake this is an old issue that should probably have been closed - can you provide your versions for everything, esp. Python and Gunicorn, and some recorded observations and logs?

I am facing this issues too. python3.8.9 gunicorn==20.1.0 Django==3.2.10

This issue occurs after I upload a large file.

[benoitc]

closiing in favour of #3250