HWM on recv side

[DrJimFan]

I've read a few similar questions before, but I'm still a bit confused by the concept of HWM on recv side. Specially, I want to use PUSH/PULL pattern so that the client's send will be blocked until server has processed enough items to make room for more incoming items in the message queue (limited by HWM).

Server code:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.set_hwm(1)
socket.bind("tcp://127.0.0.1:{}".format(6000))

print(socket.recv())
print(socket.recv())
while True:
    time.sleep(10)   # recv queue should be clogged by now

Client code:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.set_hwm(1)
socket.connect("tcp://127.0.0.1:{}".format(6000))

while True:
    socket.send(b"hello")
    print('sent')
    time.sleep(0.5)

I expect the client to be stalled after a few iterations because HWM is very low on both sides. However, it keeps printing sent instead of blocking.

[minrk]

As described in #1100, HWM cannot be used to do this. If you want the logic to depend on the server's actual processing, then you should use DEALER/ROUTER and send replies when a message has been processed and use that to throttle on the sending side. HWM cannot be used for this level of application control code. It should only be used to govern error-conditions of too-much traffic to handle. You could do something like:

outstanding = 0
max_outstanding = 3
socket = context.socket(zmq.DEALER)

while True:
    
    while outstanding >= max_outstanding:
        # wait for reply, don't send
        reply = socket.recv_multipart()
        outstanding -= 1

    socket.send(request)
    outstanding += 1

[DrJimFan]

Thanks for your help. I'll give it a try. In this case, maybe Redis is a better solution? Redis queues keep everything transparent. If sender and receiver speeds are not balanced, I'm not sure whether Zmq drops data silently in PUSH/PULL mode, which would be unacceptable in my application.