czmq: zstr_recv_nowait/1 is missing some messages

[benoitc]

from time to time czmq:zstr_recv_nowait/1 is missing some messages even if you retry it in loop. Is this expected?

[gar1t]

I'm not sure. As the current implementation just passed through calls to the port, I wonder if this is something in the zmq layer itself. Is there a special condition that causes this to manifest? Is it possible to recreate with a sample app?

[benoitc]

Good call! I forgot to provide it... I will do that until monday.

[benoitc]

hrm seems i am only able to reproduce it in the erlang shell. Using the following client in Python aith the receiver in erlang doesn't show any message mising. This ticket could probably be closed.

test_issue25.py (client):

import zmq
import time

ctx = zmq.Context()
s = ctx.socket(zmq.DEALER)
s.setsockopt(zmq.IDENTITY,b"a")
s.connect("tcp://127.0.0.1:5555")

i = 0
while True:
    msg = b"hello " + bytes(str(i))
    try:
        s.send(msg)
        print(msg)
    except KeyboardInterrupt:
        break

    i += 1
    time.sleep(0.2)

s.close()

test_issue25.erl (server):

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin
%%

main(_) ->
    {ok, Ctx} = czmq:start_link(),

    Router = czmq:zsocket_new(Ctx, router),
    {ok, 5555} = czmq:zsocket_bind(Router, "tcp://127.0.0.1:5555"),
    do_loop(Router).


do_loop(Router) ->
    case czmq:zstr_recv_nowait(Router) of
        {ok, Msg} ->
            io:format("got ~p~n", [Msg]);
        _ ->
            ok
    end,
    do_loop(Router).

[gar1t]

I'd like to poke around a bit more on this though, out of curiosity. How are you running the problem case in the shell? Is there some test code I can use to see if I can recreate this?

[benoitc]

In the shell i am just running the 3 first lines:

{ok, Ctx} = czmq:start_link(),
Router = czmq:zsocket_new(Ctx, router),
{ok, 5555} = czmq:zsocket_bind(Router, "tcp://127.0.0.1:5555").

and then manually trigger: czmq:zstr_recv_nowait(Router). which works only from time to time. Maybe a question of timing.

[gar1t]

How are you running the sender then? Assuming this is that Python loop, when do you start the Python process relative to your steps in the shell?

And given that you're sending 5 messages per second from the Python process, how do you know the messages are being dropped, versus just being received in an unexpected order? E.g. if you're just manually running receives from the Erlang shell, you might be seeing something like: 1, 2, 4, 5, 6, 3, 7, 9, 10, 8...

[benoitc]

i am just looking at the screen :)

On the shell, I bind the socket on the erlang shell , then connect to it using a dealer socket in a python shell. Then I send from the python shell and receive on the erlang shell. Most of the time I get the error atom in the erlang shell.

[kristinn]

I seem to be experiencing a similar issue. I'm using czmq:zframe_recv_all/1 and it seems to periodically loose messages.

I haven't been able to reproduce the issue using @benoitc scripts (it's not using the same zmq receiver, but they're both non blocking).

This is very random. Not sure exactly how to describe it any better than @benoitc has done already.

[kristinn]

I've been doing more throughout testing today and it seems like the issue was more related to timing out websockets and perhaps the ordering of messages. :-)