nginx freezing

[alexbirkett]

Hi,

Since I've recompiled nginx (nginx-1.2.5) with the nginx_tcp_proxy_module (latest snapshot form github), the server has started freezing. It stopps accepting http requests. The web browsers are just left waiting. Restarting nginx fixes the problem. I'm running Ubuntu 12.04.1 LTS.

Here is the error log: 2012/12/11 11:35:26 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:26 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:26 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:29 [error] 19566#0: *1 upstream servers are busy or encounter error! while connecting to upstream, client: 195.159.164.228, server: 0.0.0.0:80 2012/12/11 11:35:30 [error] 19566#0: *3 upstream servers are busy or encounter error! while connecting to upstream, client: 123.49.59.22, server: 0.0.0.0:80 2012/12/11 11:35:30 [error] 19566#0: *4 upstream servers are busy or encounter error! while connecting to upstream, client: 123.49.59.22, server: 0.0.0.0:80

The server receives a low volume of trafic. Do you know what is leaving file handles open and or why?

This may or may not be an issue with nginx_tcp_proxy_module.

Thanks,

Alex

[yaoweibin]

How many worker_connections do you set? What does this command show: ulimit -n ?

[alexbirkett]

ulimit -n returns 1024

nginx.conf contains:

events {
        worker_connections  1024;
}

[yaoweibin]

You should enlarge this two numbers. It's not suitable for production server. For example, you could specify them like this: ulimit -n 65536 worker_connections 10240

Maybe it's not enough to change these two value. You need change open file number the /etc/security/limits.conf for the owner of nginx worker.

[alexbirkett]

Yes I agree, I need to increase the maximum number of open file descriptors. My concern is that the descriptors are leaking, there are never 1024 simultaneously connected users. Could it be that if I increase the numbers, the same problem will occur but after a much greater period of time?

[yaoweibin]

On 2012-12-11 22:49, Alex Birkett wrote:

Yes I agree, I need to increase the maximum number of open file descriptors. My concern is that the descriptors are leaking, there are never 1024 simultaneously connected users. Could it be that if I increase the numbers, the same problem will occur but after a much greater period of time?

I can't guarantee it. If that happens, you could report me again.

Note that you use tcp proxy which will open two connections for one user. One for the frontend client, one for backend server.

Thanks.

Thanks, -Weibin Yao

[alexbirkett]

Ok. Let's do that. Thanks very much for your help. Please close the issue.

[yaoweibin]

You can close it yourself. let's wait for a while, hope it can solve your problem.