predixy CLOSE

由于 https://github.com/openresty/lua-resty-redis openresty/lua-resty-redis，不支持cluster模式，所以我用predixy代理。当我在压测nginx性能时，predixy出现大量CLOSE_WAITE请求未关闭情况。导致nginx连接不上predixy。有可能是我哪里配置不对嘛，或者predixy有bug。

predixy ：1.0.5 release

cluster config:

ClusterServerPool {
    MasterReadPriority 60
    StaticSlaveReadPriority 50
    DynamicSlaveReadPriority 50
    RefreshInterval 1
    ServerFailureLimit 10
    ServerRetryTimeout 1
    Password "xxx"
    Servers {
        + xxx:xxx
        + xxx:xxx
        + xxx:xxx
    }
}

/etc/sysctl.conf

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
###pvdriver<begin>
net.ipv4.conf.all.arp_notify = 1
net.ipv4.conf.default.arp_notify = 1
net.ipv4.conf.eth0.arp_notify = 1
net.ipv4.conf.lo.arp_notify = 1
###pvdriver<end>

vm.overcommit_memory = 1

Nov 04 '19 06:11 Toifyx

我的nginx配置了两个代理，图片展示的为当前服务器predixy连接CLOSE_WAIT的数量。当我依次kill掉predixy的进程的时候。连接数量由1000->500 ->2 . (剩余的两个为其他机器的连接 predixy_iss )

Nov 04 '19 07:11 Toifyx

我将集群redis换成单点redis。测试结果。并未出现CLOSE_WAIT的情况。

压测连接慢慢上来后，最大到992，压测结束后连接变成0

Nov 04 '19 08:11 Toifyx

predix日志记录的什么

Nov 04 '19 14:11 fortrue

predix日志记录的什么

你好，我找到部分日志有异常的如下：

2019-11-04 15:00:26.908966 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909023 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909089 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909134 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909134 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909184 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909223 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909269 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909272 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909327 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909361 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909378 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909514 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909522 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909574 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909779 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909797 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909815 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909833 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909852 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909872 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909887 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909907 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909920 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909942 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.909955 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.910008 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911019 N Handler.cpp:303 h 2 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911389 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911420 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911442 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911480 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911486 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911524 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911524 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911572 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911652 N Handler.cpp:303 h 0 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911763 N Handler.cpp:303 h 3 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911573 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911844 N Handler.cpp:303 h 2 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files
2019-11-04 15:00:26.911869 N Handler.cpp:303 h 1 p 0.0.0.0:7617 3 e ListenSocket.cpp:50 socket accept fail Too many open files

这看起来有点像ListenSocket.cpp，这个文件throw的异常，是否是在这里没有close连接。

/*
 * predixy - A high performance and full features proxy for redis.
 * Copyright (C) 2017 Joyield, Inc. <[email protected]>
 * All rights reserved.
 */

#include <string.h>
#include "Util.h"
#include "ListenSocket.h"

ListenSocket::ListenSocket(const char* addr, int type, int protocol)
{
    mClassType = ListenType;
    strncpy(mAddr, addr, sizeof(mAddr));
    sockaddr_storage saddr;
    socklen_t len = sizeof(saddr);
    getFirstAddr(addr, type, protocol, (sockaddr*)&saddr, &len);
    sockaddr* in = (sockaddr*)&saddr;
    int fd = Socket::socket(in->sa_family, type, protocol);
    int flags = 1;
    setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags));
    int ret = ::bind(fd, in, len);
    if (ret != 0) {
        int err = errno;
        ::close(fd);
        errno = err;
        Throw(BindFail, "socket bind to %s fail %s", addr, StrError());
    }
    attach(fd);
}

void ListenSocket::listen(int backlog)
{
    int ret = ::listen(fd(), backlog);
    if (ret != 0) {
        Throw(ListenFail, "socket listen fail %s", StrError());
    }
}

int ListenSocket::accept(sockaddr* addr, socklen_t* len)
{
    while (true) {
        int c = ::accept(fd(), addr, len);
        if (c < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                return -1;
            } else if (errno == EINTR || errno == ECONNABORTED) {
                continue;
            } else if (errno == EMFILE || errno == ENFILE) {
                Throw(TooManyOpenFiles, "socket accept fail %s", StrError());
            }
            Throw(AcceptFail, "socket accept fail %s", StrError());
        }
        return c;
    }
    return -1;
}

其他日志。这里就不贴出来了（看起来应该是正常的输出日志）

Nov 05 '19 01:11 Toifyx

我也碰到了连接数过多的问题，

Aug 30 '21 07:08 akin520

。这里就不贴出来了（看起来应该是

你好，这个问题你解决了么？

Aug 03 '22 07:08 wyl9527

我也碰到了连接数过多的问题，

你好这个问题，你解决了么？

Aug 03 '22 07:08 wyl9527

。这里就不贴出来了（看起来应该是

你好，这个问题你解决了么？

没有，换其他方式了。没有使用代理

Aug 26 '22 03:08 Toifyx

socket accept fail Too many open files的的报错，大概是文件句柄不够 问题查看： 用ulimit -a查看当前的文件句柄限制（最大65535）再看下进程占用的文件句柄数，是否达到了限制 lsof -p 代理predixy的pid | wc -l

解决办法： vim /etc/security/limits.conf
#在最后加入

* soft nofile 4096  
* hard nofile 4096

再重启下代理进程即可

Nov 08 '22 07:11 Foliertovel

predixy predixy copied to clipboard

CLOSE_WAITE

predixy
predixy copied to clipboard