Sanitizer Errors When Zmq Send/Recv Structs with std::variant

[esaliya]

I've a simple struct similar to the following that I want to send/recv using cppzmq.

struct SimpleSt{
    explicit SimpleSt(int num): var(num){}
    explicit SimpleSt(std::string str) : var(str){};
    std::variant<int, std::string> var;
};

This is my main method.

int main()
{
    SimpleSt msg(33);

    // Creates a zmq message out of msg
    zmq::message_t zmqMsg(&msg, sizeof(msg));

    // Creates a client/server socket pair to test
    zmq::context_t ctx{1};
    zmq::socket_t  csock(ctx, ZMQ_PAIR);
    zmq::socket_t  ssock(ctx, ZMQ_PAIR);
    ssock.bind("tcp://*:9090");
    csock.connect("tcp://localhost:9090");

    // Client socket send to server
    csock.send(zmqMsg, zmq::send_flags::none);

    // Receive
    zmq::message_t recvdZmqMsg;
    auto           rc = ssock.recv(recvdZmqMsg, zmq::recv_flags::none);
    assert(rc >= 0);

    std::cout << std::get<0>(recvdZmqMsg.data<SimpleSt>()->var) << std::endl;
}

But when I run this, I get a bunch of sanitizer errors as shown.

Appreciate any help with this.

[gummif]

This is unsafe since you are interpreting bytes as an object. It is also pobably not a good idea to send the bytes representing variant since it is not a trivial type, and it contains a string which might be allocated on the heap, so it wouldn't work anyway (between processes or computers).

I would suggest serializing the data as json or some other standard format.

[esaliya]

@gummif so is get<T>() only recommended for trivial types? I further simplified this bug that happens without variants or strings. For example, just sending the following struct would cause misaligned errors because the receive buffer is not arbitrarily aligned.

struct Packet
{
    uint64_t arr[4];
    uint32_t id;
};

What's the proper way to send and receive a struct like this with zmq?

[gummif]

The data<>() function is really only safe for byte like types, unsigned char, char, and std byte. That should probably be better documented.

But if you are sending a trivial struct like that then the recommended way to get an object from some bytes is by memcpy. Create an object of type Packet and then memcpy the data() from the message into the object. Here are some more details https://en.cppreference.com/w/cpp/numeric/bit_cast

[esaliya]

So is there a way we can provide a pre-allocated buffer to zmq::message_t that'll be used during receive? There's an API for that but I found recv operation does not use that buffer, instead use its own internal buffer to receive data.

I am trying to avoid the memory copy because some other data types are expensive than the simple struct shown here.

[gummif]

I think libzmq uses messages internally so when you recv messages (heap allocated ones) that is what you get. You could check the alignment of the data pointer at runtime, but I'm not sure how safe that is.

The simplest option is to do something like

Foo foo;
s.recv(zmq::buffer(&foo, sizeof foo));

but I think that libzmq internally is just copying from the message buffer. You should probably benchmark this to determine if this sufficient.

[esaliya]

After digging into zmq code, I see that even if you provide a buffer, zmq will not avoid copying as it receives to an internal buffer and then copies it over to the given buffer.

It also produces a notasocket error.

[gummif]

ZMQ_PAIR is designed for inproc transport so maybe try that, or some other socket type over tcp.

[sigiesec]

Some comments:

• Re message_t::data<T>: we should probably std::enable_if it only when T is std::is_pod<T>::value is true? But that still doesn't solve an issue with alignment.
• Re alignment: that's a libzmq question. It has been asked before, e.g. in zeromq/libzmq#3316. You can get word-aligned buffers when disabling zero-copy, but as said there, this is probably not desirable.
• Also a libzmq question, but I don't think that it's accurate to say that ZMQ_PAIR is designed for inproc. I don't see any reason why it shouldn't work with tcp transport.

[gummif]

According to the docs tcp for pair is not suitable in some cases http://api.zeromq.org/4-3:zmq-socket but that may not be the case here.

[sigiesec]

According to the docs tcp for pair is not suitable in some cases http://api.zeromq.org/4-3:zmq-socket but that may not be the case here.

Sorry, my fault. You're actually right, that's what the documentation says. I am not sure what I thought exactly. (I think it's not relevant to the problem here though)