hiredis icon indicating copy to clipboard operation
hiredis copied to clipboard

Problem when sending messages when TLS encrypted connection is used

Open R-15000 opened this issue 3 months ago • 6 comments

Hello,

We are doing speed and load tests using Redis in pub/sub broker mode and Hiredis C API. The tests consists of continuously sending messages both ways from a client A to a client B using the broker.

We have an issue when sending messages when TLS encrypted connection is used.

Regarding our tests :

  • We have a client A which sends a message to the client B. The client B, after receiving the message from client A, replies to the client A from the message reception callback. We repeat this operation continuously to mesure the number of transactions per second.
  • To do so, we use a topic "client_A" for the client A to send the messages, the client B subscibes to this topic "client_A". Also, the client B sends the messages to its own topic "client_B", the client A subscribes to this topic "client_B".

When we don't use TLS encrypted connection, everything works just fine. But when we set TLS encrypted connection, the message reception seems ok but seems we have issues sending some messages. In our case, when TLS is set, the client B still receives the message from the client A, then the receive callback is called normally, but when it sends the reply from the callback, it seems the reply is not sent.

We found a temporary way to bypass, in our client B code, in the main function, before the test starts, we add the following lines to send a message to a temporary topic (Used only once), then add 1 second sleep delay after calling event_base_loop :

redisAsyncCommand(Redis_async_context_reply, callback, (char *) "pub", "Publish Temporary_topic Hello", 5); event_base_loop(Base_reply, EVLOOP_NONBLOCK); sleep(1);

If we do this bypass, we are able to send the reply from the client B callback and the test seems to work fine.

However, we face strange limitations and restrictions with this bypass : If we send the "Hello" message to the topic "client_A" (that is used by client B to send replies from the callback), insted of a temporary topic, it doesn't work in that case. Also, when we bypass, in the main function, before the test starts, if we send another message on another topic, no matter which, after doing the bypass (by sending the Hello message to the temporary topic), it doesn't work either.

Technically, there is also a second bypass, if we add a sleep(1) in the client B callback, after calling redisAsyncCommand and event_base_loop, it works, but it is of course painfully slow (1 message per second only). So we cannot use this method.

If we don't use TLS encrypted connection, everything works fine, we don't need to bypass.

We are trying to find the origin of this issue which appears only when we use TLS in our case. We guess there is somewhere an additional delay with the first message or something like that, anyway something that causes perturbation with event_base_loop or maybe redisAsyncCommand. We tried to check the result of redisAsyncCommand before calling event_base_loop, and the same for event_base_loop, there is no error returned. (However we are not sure this verification is 100% reliable, as we got issues with the API of another broker : The publish function return was ok, meaning the command was sent, but the message sending was still in progress for example.)

In the client B (and also the client A) program, we use 2 Hiredis contexts, one for message reception, one for message sending. In the main function, when the test is started, we call, in a while loop, event_base_loop for the reception context. We also tried to call event_base_loop for the message sending context in the main loop instead of calling it only inside the callback after a message is sent, but it doesn't help, we still have the issue.

Any help would be appreciated.

Thank you.

R-15000 avatar Mar 27 '24 13:03 R-15000