ManagedFastCgiListener not closing sockets

[kcd83]

Hello. Great work with the project. Unfortunately I am having issues with the ManagedFastCgiListener under CentOS 6.

Have you seen this problem where many sockets are not closing?

I can reproduce with ab -t60 -n 50000 -c 50 'http://somehost' and watching the sockets/file descriptors climb (netstat -ap | grep mono | wc -l and ls -l /proc/pidof mono/fd) until hyperfastcgi hits the ulimit and crashes.

unix  3      [ ]         STREAM     CONNECTED     1967223 21956/mono          /tmp/hyperfastcgi.socket
unix  3      [ ]         STREAM     CONNECTED     1967221 21956/mono          /tmp/hyperfastcgi.socket
unix  3      [ ]         STREAM     CONNECTED     1967219 21956/mono          /tmp/hyperfastcgi.socket
unix  2      [ ]         STREAM     CONNECTED     1967215 21956/mono          /tmp/hyperfastcgi.socket
unix  2      [ ]         STREAM     CONNECTED     1967209 21956/mono          /tmp/hyperfastcgi.socket
unix  3      [ ]         STREAM     CONNECTED     1966980 21956/mono          /tmp/hyperfastcgi.socket

hyperfastcgi

<configuration>
    <server type="HyperFastCgi.ApplicationServers.SimpleApplicationServer">
        <host-factory>HyperFastCgi.HostFactories.SystemWebHostFactory</host-factory>
        <threads min-worker="40" max-worker="0" min-io="4" max-io="0" />
    </server>
    <listener type="HyperFastCgi.Listeners.ManagedFastCgiListener">
        <listener-transport type="HyperFastCgi.Transports.ManagedFastCgiListenerTransport"></listener-transport>
        <apphost-transport type="HyperFastCgi.Transports.ManagedAppHostTransport"></apphost-transport>
        <protocol>Unix</protocol>
        <address>/tmp/hyperfastcgi.socket</address>
    </listener>
    <apphost type="HyperFastCgi.AppHosts.AspNet.AspNetApplicationHost">
        <log level="Debug" write-to-console="true" />
        <add-trailing-slash>false</add-trailing-slash>
    </apphost>
    <web-applications>
        <web-application>
            <name>website</name>
            <vhost></vhost>
            <vport>80</vport>
            <vpath>/</vpath>
            <path>/var/www/website</path>
        </web-application>
    </web-applications>
</configuration>

nginx

upstream hyperfastcgi_backend {
  server unix:/tmp/hyperfastcgi.socket;
  keepalive 32;
}
server {
  listen 80;

  client_max_body_size 50M;
#  keepalive_timeout 5;

  access_log /var/log/nginx/access.log;

  location / {
#    proxy_read_timeout 600;
    root /var/www/website;
    index default.aspx Default.aspx;
    fastcgi_index Default.aspx;
    fastcgi_keep_conn on;
    fastcgi_pass hyperfastcgi_backend;
    include /etc/nginx/fastcgi_params;
  }
}

mono is a custom build of 3.2 which is not the newest so I haven't ruled that out yet.

[xplicit]

Do you see this issue only with managed listener? Did you try NativeListener?

With managed listener: what if you change keepalive settings in nginx to 60? What is the state of sockets when number of sockets are close to maximal allowed?

[kcd83]

For the NativeListener I need libevent2 which I am avoiding due to a conflict we had previously. I could revisit this.

All connections claim to be connected and I got up to 64k of them. When generating traffic I do see some connections come and go but most stay open forever.

Yes I can try 60s keepalive On 17/03/2015 8:32 PM, "Sergey Zhukov" [email protected] wrote:

Do you see this issue only with managed listener? Did you try NativeListener?

With managed listener: what if you change keepalive settings in nginx to 60? What is the state of sockets when number of sockets are close to maximal allowed?

— Reply to this email directly or view it on GitHub https://github.com/xplicit/HyperFastCgi/issues/23#issuecomment-82182847.

[xplicit]

With keepalive 60 and ab concurrent connections 50 nginx should not create any additional connections so the number of opened connection should not grow. If it grows it is probably something wrong with nginx installation. If it does not grow then the issue on HFC/mono side. I did not see such issue on Ubuntu system, but I'll recheck it.

[kcd83]

Ah I see what is happening now thanks. With that scenario you are correct I have no problem.

However I read up on this:

It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open. The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well.

• http://nginx.org/en/docs/http/ngx_http_upstream_module.html

Obviously my test was an extreme example but the occasional extra connection nginx opens is never closed (fcgi_keep_conn on). In a long running system under load will eventually suffer the same problem. Also if nginx is restarted often. If I can find time tomorrow I will try some more things out.

[kcd83]

A quick update. I toyed with some changes (hacks) to clean up stale connections with some success but moved on with performance testing using the higher number of keepalive sockets workaround above.

Using a small vagrant vm and apache bench i get:

helloworld equivalent - something static

• 900 r/s unix keepalive off
• 900 r/s unix keepalive 200
• 900 r/s tcp keepalive off
• 900 r/s tcp keepalive 200

in-memory-response - something cpu bound with no other i/o like database

• 275 r/s unix keepalive off
• 230 r/s unix keepalive 200
• 275 r/s tcp keepalive off
• 230 r/s tcp keepalive 200

So perhaps the better solution (when nginx and hyperfastcgi are co-located) is to turn off keepalives.

As for this bug report, I feel like this is an nginx issue where by nginx should be configurable to not send fastcgi_keep_conn on for sockets greater than keepalive. If it wasn't for the above results I was hoping to try some other fastcgi implementation as a comparison.

On a wild tangent, be very wary of the default MONO_EVENTLOG_TYPE="local:/var/log/somewhere" ... in one scenario i was logging process times and the above dropped to just 35 r/s ...!