ngx.exit() closes the connection. so benchmark tools with socket reuse (keep alive) do not work

[remort]

trafficstars

Hi! I perform a load testing for my web-server. I use wrk since I find it more suitable for loading up the network and the target web-server with a traffic, hence getting a more truthful results.

In several cases web-server's nginx location uses lua scripts. This scripts returns 403 (ngx.exit(ngx.HTTP_FORBIDDEN)). In these cases wrk hangs, stopping sending requests after several seconds. The number of open sockets on web-server from a client becomes 0. And it looks that wrk is unable to reuse closed sockets. Without lua scripting (just nginx' proxy_pass) return of 200 and 403 is possible without closing the socket so wrk or other benchmarks using keep alive continue to work. Siege (or ab) benchmark works fine being configured to not to reuse sockets but rather close them after every request. Closing sockets after every request is slow and can lead to out of sockets problem on either client or server. You need to deal with TIME_WAIT, CLOSE_WAIT hanging sockets and so on. And I want my web-server to be keep-alive compliant anyway.

Is there an alternative way to finish current TCP-session without closing the network socket ?

[piotrp]

You can increase the number of available sockets by changing net.ipv4.ip_local_port_range, and allow for socket reuse by setting net.ipv4.tcp_tw_reuse = 1 (both are sysctl parameters). Also, keep in mind that unless you set -H "Connection: close" in wrk nginx will reuse connections due to keepalive_requests.

[remort]

Hi, @piotrp . I know all this. One doesn't play with default sysctl settings in production environment. wrk and nginx work well and I allow them to reuse sockets. It all goes well until lua's ngx.exit. wrk is unable to reuse sockets in this case. So I conclude that ngx.exit closes socket or affect keep alive/socket reuse mechanics in some other way.

[remort]

I can reproduce it.

1. We have an nginx with location containing lua code.
2. traffic goes to aiohttp upstream using upstream_response = ngx.location.capture(...) in access_by_lua_block, then it comes back to this lua blocks.
3. a var is being set, say ngx.var.res = upstream_response.header["X-Result"] to be able to access to it in the next lua block.
4. in the next header_filter_by_lua_block block I analyze this var. If the result is positive, I must to send 403 Forbidden, otherwise 200 OK to the client. I do it with the following code:

    header_filter_by_lua_block {
      if ngx.var.res then
        ngx.exit(ngx.HTTP_FORBIDDEN)
      else
        ngx.exit(ngx.HTTP_OK)
      end
    }

Everything works fine (curl, browser, curl run in a bash loop, -H "Connection: close/keep-alive") until I benchmark nginx (with this location with proxy_pass and ngx.exit) wrk2. wrk2 has a -c parameter. Number of simul. connections. There will be sent exactly N requests by wrk2 if I use nginx lua with this force connection closes (ngx.exit).

wrk2 sends packets continuously if I omit ngx.exit(ngx.HTTP_OK). But then I get 404 error from nginx because nginx location serves no URLs except upstream location via lua scripting. So I must to skip nginx response stage and send the 200/403 response via lua, but lua does it wrong, causing wrk2 to stop sending requests after first one per connection in http1.1 mode (when wrk2 sends Connection: keep-alive and sockets in a number of -c N are being reused).

So:

• with ngx.exit() wrk2 sends requests only in http1.0 mode, i.e. "Connection: close". otherwise it stops sending requests after first one inside the connection. I.e. 10 connections - 10 requests.
• without ngx.exit() wrk2 sends requests with no issues in any mode: http1.0 or http1.1, succesfully reusing sockets in case of http1.1. So NGINX's responses (as 404) are ok with wrk2.

Will test it a bit to provide more info. But you already can reproduce it.

[remort]

Ok, I found it. My first block with ngx.location.capture is access_by_lua_block. My second lua block, where I was calling ngx.exit() was header_filter_by_lua_block. Problem has gone when I changed the name of the block to content_by_lua_block. So ngx.exit() differently for different stages. And according to this document content_by_lua_block is an earlier stage than header_filter_by_lua_block.

May be wrk2 can't reuse connection being finished on certain stages.