dpvs icon indicating copy to clipboard operation
dpvs copied to clipboard

Published 20 hours ago • verified publisher icon iqiyi

udp的fullnat模式 nginx syslog 下rs 连接不会释放

Open snailq opened this issue 5 years ago • 3 comments

client (ngin开启rsyslog) ---> vip (dpvs udp fullnat) ---> rs（rsyslog server）

ipvsadm -ln
UDP  10.136.102.100:514 wrr
  -> 10.136.27.112:514            FullNat 100    0          4
  -> 10.136.27.113:514            FullNat 100    0          4

下掉一台rs（rsyslog）

ipvsadm -ln
UDP  10.136.102.100:514 wrr
  -> 10.136.27.112:514            FullNat 100    0          4

下掉的rs 10.136.27.113 连接不释放，rs上抓包还一直收到dpvs转发过来的流量，即使把这台rs down掉，dpvs还会不停的转发数据到这台rs上，只要nginx不停的发数据过来，停掉nginx等10s释放掉再起才能释放已有的连接

至少等待1小时以上连接也不会释放 只要nginx端不停止发数据 不重新建立连接就不会释放

ipvsadm -lnc| grep 10.136.27.113
[1]udp 10s    UDP_NORM    10.136.27.210:21938 10.136.102.100:514 10.136.58.16:1032  10.136.27.113:514
[1]udp 10s    UDP_NORM    10.136.27.210:38191 10.136.102.100:514 10.136.58.12:1032  10.136.27.113:514
[1]udp 10s    UDP_NORM    10.136.27.211:13927 10.136.102.100:514 10.136.58.14:1032  10.136.27.113:514
[1]udp 10s    UDP_NORM    10.136.27.210:36377 10.136.102.100:514 10.136.58.10:1032  10.136.27.113:514

原因是nginx 客户端发往vip的源ip和源端口不变，dpvs会保持这个连接等待空闲释放，但是nginx syslog使用udp模式不会检测是否发过去的数据是否失败 只会不停的发，导致不会重新 新建连接，dpvs应该去检测rs存活状态强制停止这个连接才对啊

在9月份之前的dpvs源码测试不会有这个问题，只会有连接不均匀，这个可以理解因为client的源ip和源端口不变，只能均衡新建的连接，已有连接 再不断开的情况下没办法均衡

snailq avatar Jan 12 '19 10:01 snailq 

UDP  10.136.102.100:514 wrr
  -> 10.136.27.112:514            FullNat 100    0          4
  -> 10.136.27.113:514            FullNat 100    0          4
ipvsadm -lnc
[1]udp 10s    UDP_NORM    10.136.27.210:7202 10.136.102.100:514 10.136.58.10:1032  10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.211:57097 10.136.102.100:514 10.136.58.2:1026   10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.211:13927 10.136.102.100:514 10.136.58.7:1026   10.136.27.113:514
[3]udp 10s    UDP_NORM    10.136.27.211:58426 10.136.102.100:514 10.136.58.4:1026   10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.210:61339 10.136.102.100:514 10.136.58.3:1026   10.136.27.113:514
[4]udp 10s    UDP_NORM    10.136.27.211:36814 10.136.102.100:514 10.136.58.9:1027   10.136.27.113:514
[4]udp 10s    UDP_NORM    10.136.27.210:30849 10.136.102.100:514 10.136.58.6:1027   10.136.27.112:514
[6]udp 10s    UDP_NORM    10.136.27.210:26398 10.136.102.100:514 10.136.58.5:1029   10.136.27.113:514


ipvsadm -d -u 10.136.102.100:514 -r 10.136.27.113

UDP  10.136.102.100:514 wrr
  -> 10.136.27.112:514            FullNat 100    0          4

 ipvsadm -lnc
[1]udp 10s    UDP_NORM    10.136.27.210:7202 10.136.102.100:514 10.136.58.10:1032  10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.211:57097 10.136.102.100:514 10.136.58.2:1026   10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.211:13927 10.136.102.100:514 10.136.58.7:1026   10.136.27.113:514
[3]udp 10s    UDP_NORM    10.136.27.211:58426 10.136.102.100:514 10.136.58.4:1026   10.136.27.112:514
[3]udp 10s    UDP_NORM    10.136.27.210:61339 10.136.102.100:514 10.136.58.3:1026   10.136.27.113:514
[4]udp 10s    UDP_NORM    10.136.27.211:36814 10.136.102.100:514 10.136.58.9:1027   10.136.27.113:514
[4]udp 10s    UDP_NORM    10.136.27.210:30849 10.136.102.100:514 10.136.58.6:1027   10.136.27.112:514
[6]udp 10s    UDP_NORM    10.136.27.210:26398 10.136.102.100:514 10.136.58.5:1029   10.136.27.113:514

snailq avatar Jan 14 '19 08:01 snailq

RS上看一下UOA有没有安装对应版本。另外，dpvs/lvs这类四层负载均衡器，不会因为健康检查不通过强制断开已有连接。健康检查是对后续新建连接的调度起作用，对已有连接不起作用。已有连接要么靠状态机关闭，要么靠超时关闭。UDP没有状态机，只能靠超时，而端上如果一直有流量是无法释放的。UDP的设计初衷就是要靠用户自己去管理session，需要上层应用自己去感知连接无效/超时，自己去关闭、新建连接。

mscbg avatar Jan 22 '19 02:01 mscbg

没有安装UOA，因为实际测试，如果不配置uoa_max_trail=0 总会有问题，前三个包总会丢失，换了几种udp的服务都是一样。

另外udp连接不释放的问题，lts版本没这个问题，只有master和devel版本有这种问题

snailq avatar Jan 31 '19 03:01 snailq