REP socket memory leak

[franzhollerer]

Issue description

We use the REP/REQ socket pattern and faced a memory leak at the server side which implements the REP part. It revealed by "accident", but we think it is a bug in the libzmq library which should be fixed.

At server side we created a REP socket and bound it to an ipc:// endpoint. Unfortunately, the receiving thread in the server was buggy, and did not read from the socket. Our client sent a message to the server, and as the client did not get a response it closed the socket, reconnected and repeated the request.

We found that the memory consumption steadily increases, although ZMQ_RCVHWM and ZMQ_SNDHWM are set. After some time the OOM-Killer of the Linux Kernel terminated the server process.

We were able to strip down the problem to a minimum working example. Find below the code for the server zmq_memory_eater_server.cpp, and for the client zmq_memory_eater_client.cpp. Compile server and client and start them. You will find that the memory consumption of the server steadily increases.

zmq_memory_eater_server.cpp:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <zmq.hpp>

const char* TEST_ENDPOINT = "ipc:///tmp/zmq_memory_eater.ipc";

int main(int argc, char** argv)
{
    zmq::context_t zmq_ctx;

    printf("Run as server (idle loop).\n");
    zmq::socket_t socket = zmq::socket_t(zmq_ctx, zmq::socket_type::rep);
    socket.set(zmq::sockopt::rcvhwm, 100);
    socket.set(zmq::sockopt::sndhwm, 100);
    socket.bind(TEST_ENDPOINT);
    while (1) {
        sleep(1);
    }

    return 0;
}

zmq_memory_eater_client.cpp:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <stdint.h>
#include <vector>
#include <algorithm>
#include <chrono>
#include <zmq.hpp>

const char* TEST_ENDPOINT = "ipc:///tmp/zmq_memory_eater.ipc";

int main(int argc, char** argv)
{
    zmq::context_t zmq_ctx;

    std::vector<uint8_t> msg(1024);
    std::generate(msg.begin(), msg.end(), std::rand);

    printf("Run as client (sending loop).\n");
    for (;;) {
        // (re-)connect
        zmq::socket_t socket = zmq::socket_t(zmq_ctx, zmq::socket_type::req);
        socket.connect(TEST_ENDPOINT);

        // try to send message
        if (!socket.send(zmq::message_t(msg.data(), msg.size()),
                    zmq::send_flags::none)) {
            printf("couldn't send message\n");
        }

        // run into timeout (100ms)
        zmq::pollitem_t in = {socket, 0, ZMQ_POLLIN, 0};
        auto rc = zmq::poll(&in, 1, std::chrono::milliseconds(100));
        assert(rc == 0);
    }

    return 0;
}

Environment

• libzmq commit hash: a01d259db372bff5e049aa966da4efce7259af67
• cppzmq commit hash: a98fa4a91d868a3844e5456741d6782cc1a8d98b
• OS: Ubuntu 20.04.3 LTS

Minimal test code / Steps to reproduce the issue

1. start zmq_memory_eater_server
2. start zmq_memory_eater_client
3. use top or htop to watch the steady increasing memory consumption of zmq_memory_eater_server

What's the actual result? (include assertion message & call stack if applicable)

The memory consumption of zmq_memory_eater_server steadily increases.

What's the expected result?

There should be an upper limit for the used memory, especially as the socket option ZMQ_RCVHWM is set to a quite low level. libzmq should drop new messages after reaching the high water mark.

[Chekov2k]

We have just found the exact same behaviour in our code using Req/Rep with czmq. This behaviour should be a big, shouldn’t it?

[Chekov2k]

Having read more and looking at the description of the ZMQ_CONFLATE socket options, it seems it is expected behaviour that the socket just accepts messages under the hood until they are are received

[franzhollerer]

Well, it might have been planned this way. But I personally think it is not a sane behavior. Especially, when I set ZMQ_RCVHWM and ZMQ_SNDHWM I expect that there is a reasonable limitation of memory consumption, and that messages get dropped when the limit is reached rather than get killed by the OS.

[Chekov2k]

The documentation should be more explicit about that behaviour, I agree