predixy icon indicating copy to clipboard operation
predixy copied to clipboard

fix bug : leak private connections

Open ikenchina opened this issue 1 year ago • 0 comments

Handler::postConnectConnectionEvent() function put idle connection to pool twice.

pseudocode

void Handler::postConnectConnectionEvent()
{
    while (!mPostConnectConns.empty()) {
        ConnectConnection* s = mPostConnectConns.pop_front();
        ......
        if ((!s->good()||(evts & Multiplexor::ErrorEvent)) && s->fd() >= 0){
            ......
            
            // it call putPrivateConnection first time.
            s->close(this);
            if (!s->isShared()) {
                // put pool again
                mConnPool[s->server()->id()]->putPrivateConnection(s);
            }
            ......
        }
    }
}

void ConnectConnection::close(Handler* h)
{
    SendRequestList* reqs[2] = {&mSentRequests, &mSendRequests};
    for (int i = 0; i < 2; ++i) {
        while (!reqs[i]->empty()) {
            auto req = reqs[i]->front();
            h->directResponse(req, Response::ServerConnectionClose, this);
            reqs[i]->pop_front();
        }
    }
    ......
}

void Handler::directResponse(Request* req, Response::GenericCode code, ConnectConnection* s)
{
     ......
                ResponsePtr res = ResponseAlloc::create(code);
                handleResponse(s, req, res);
      ......
}

void Handler::handleResponse(ConnectConnection* s, Request* req, Response* res)
{
    ......

    if (c->send(this, req, res)) {    }

    ......
    if (s && !s->isShared()) {
        if (!c->inTransaction() && !c->inSub(true)) {
            // problem is here
            mConnPool[s->server()->id()]->putPrivateConnection(s);
            c->detachConnectConnection();
            s->detachAcceptConnection();
        }
    }
}

reproduce steps

  • send transaction requests to predixy, about100 QPS : multi, set, set, exec
  • change redis timeout to 2 seconds : redis-cli -p 6379 config set timeout 2
  • drop redis traffics : sudo iptables -A INPUT -p tcp --dport 6379 -j DROP
  • wait 3 seconds
  • delete iptable rule : sudo iptables -D INPUT -p tcp --dport 6379 -j DROP
  • redis-cli -p 6379 config set timeout 2000

check whether connections are leaking :
watch -n 1 "netstat -apn | grep 6379 | grep ESTAB | wc -l"

ikenchina avatar Jul 19 '24 02:07 ikenchina