jeromq
jeromq copied to clipboard
zeromq
SUB socket will not receive messages if more than 1999 subscriptions are done
Hi, I have discovered in a complex application a problem when my JeroMQ SUB socket is subscribed to more than 1999 partitions: no messages get received. I managed to reproduce the issue in a 100-lines example that I'm attaching to this issue. Is there an hardcoded/configurable limit about the max number of subscriptions for SUB sockets in JeroMQ?
This is the sample application:
To run the sample, on 1st terminal run:
java -jar test_java_zmq/target/zmqtest-1-jar-with-dependencies.jar pub tcp://*:5555
and on 2nd terminal run:
java -jar test_java_zmq/target/zmqtest-1-jar-with-dependencies.jar sub tcp://*:5555
with NUM_PARTITIONS=1999 it works, with NUM_PARTITIONS=2000 it does not: no packets received at all...
(note: in the example partition=subscription)
Hi, I noticed that you commented the sending waterwark setting for the subscriber, does it change anything for you? The default is 1000 messages. Try to increase it to the maximum number of subscriptions you are sending at once, or 0 to disable it.
Hi @fredoboulo , I didn't play with the HWM in this case (I copy/pasted that commented-out portion about HWM)... indeed increasing the HWM away from the default (which I discovered is 1000) fixes the problem, thanks!
My question is: is this behavior (HWM setting applied also to PUB-SUB subscriptions messages) coherent with the C/C++ zeromq library? I don't recall having such issue with libzmq even though I don't have at hand a sample for the test...
That's a good question that would need to be confirmed by checking the behaviour. In the meantime, this comment: https://github.com/zeromq/libzmq/blob/master/src/xsub.cpp#L288 seems to indicate this is indeed coherent between jeromq and libzmq.
Hi @fredoboulo , I saw that comment but I did experimenti with a small C example (I will try to package that and link it here) and could 1048576 subscriptions on a socket having RX/TX HWM set to 1000 and then, when I later connected the PUB and run traffic, I was able to receive it... this seems to indicate that bare libzmq does not apply any HWM to subscriptions messages...
It would be useful to see such a C code.
Meanwhile, I'll PR a test demonstrating the loss of subscriptions when far exceeding HWM, together with a fix made by libzmq.
It would be useful to see such a C code.
Meanwhile, I'll PR a test demonstrating the loss of subscriptions when far exceeding HWM, together with a fix made by libzmq.
@fredoboulo Did you end up making that PR?
