Connection pool uses connection must be expired

[bobunderson]

trafficstars

We are using lua-resty-redis in high load environment, and encountered unexpected behavior of connection pool.

The webapp server communicates the kvs(Redis) server via the lvs server as below.

  WebApp <----> Lvs <----> Kvs

I show summarized lua codes as below.

  redis = require 'resty.redis'
  client = redis:new()
  client:set_timeout(5000)
  ok, err = client:connect(kvs_vip, 6379)
  ...
  (keys, set, get, hmget, etc.)
  ...
  client:set_keepalive(895*1000, 100)

Almost of requests are processed normally, but nginx writes error_log like as below sometimes.

  ... recv() failed (104: Connection reset by peer) ...

I monitored communications between WebApp and Lvs by tcpdump, and found unexpected behavior.

  (tcpdump at Lvs)
  15:02:55   <- start tcpdump
  ...
  15:48:03.085240
      webapp.60060 > kvs_vip.6379: Flags [P.], ... RESP "keys" "foo..."
  15:48:03.085283
      kvs_vip.6379 > webapp.60060: Flags [R], ...
  
  (tcpdump at WebApp)
  15:02:55   <- start tcpdump
  ...
  15:48:03.084994
      webapp.60060 > kvs_vip.6379: Flags [P.], ... RESP "keys" "foo..."
  15:48:03.085396
      kvs_vip.6379 > webapp.60060: Flags [R], ...

I found some packets between webapp.otherport and kvs_vip.6379 in time range from 15:02:55 to 15:48:03, but webapp.60060 and kvs_vip.6379 were not communicating absolutely. So, it seems like nginx used a socket which must be expired and lvs host responded to it by RST packet.

Lvs timeout parameters:

   Lvs# ipvsadm -Ln --timeout
   Timeout (tcp tcpfin udp): 900 120 300

According to "ipvsadm -Lnc" command log, ip_vs session of webapp.60060 was expired at 15:14:41.

Versions: nginx-1.14.0 lua-nginx-module-0.10.13 lua-5.1.5 lua-resty-redis-0.26