ZeroMQ isn't awesome

[arodland]

ZeroMQ has done a great job so far, but it has a few issues.

1. The client library isn't completely trustworthy, and some people (@silas) aren't comfortable with embedding libzmq in nginx.
2. A ZeroMQ bound socket can only have a single socket type. The upshot is that port numbers multiply. We already use one port number for gatekeeper->master pubsub traffic (a SUB on the master) and one for master->gatekeeper pubsub traffic (a PUB on the gatekeeper). Directed traffic between individual gatekeepers and the master would require yet another port. I don't want this to happen, so I'm making this issue a blocker of #16.
3. The more I get into it, the more the semantics of ZeroMQ aren't quite perfect. For instance, a PUB client connecting to a SUB listen socket will have all of its messages dropped until a round-trip of messages goes through to transmit the subscription(s), and there's no way to get informed of when that happens. The REQ/REP lockstep thing is also less than awesome.

I'm working on a custom UDP protocol to replace ZeroMQ, to hopefully alleviate all of these.

[ddragosd]

I'm not sure if it was any progress on this issue lately. I'm working to achieve a similar goal with nginx and lua ( via Openresty ), and I'm looking for a UDP multicast message/event system; the premise is that rate limiting doesn't have to be ultra precise but allow some overflow.

1. did you experience any issues with zeromq so far ? Is this a choice by preference that people are not comfortable embedding it ?
2. did you try running zeromq on a different IP and don't create congestion on the number of opened ports for nginx ? a new IP should give you about 64k extra connections
3. Is this problem happening when a new node is added into the game ?

[arodland]

Hi @ddragosd. No, I haven't been able to do make a ton of progress on this lately, other things have intervened. But I do have some work on a branch and I intend to get back to it soon. In answers to your questions:

1. The main problem I've encountered is that it's hard to get everything I want to do to fit within the patterns supported by ZeroMQ socket types. I have spoken to a few people who are worried about the possibility of embedding libzmq in nginx though because libzmq is peppered with asserts that could bring the server down.
2. It's not sheer number of sockets I'm worried about, it's conceptual complexity.
3. Yes, that's the current problem. When a new node comes online, or an existing one is restarted, it doesn't have information about blocks that are currently outstanding. It needs a way to get that information on startup.

[arodland]

ZeroMQ 4 looks a lot better if I want to resurrect this. In particular, ZMQ_XPUB_WELCOME_MSG could allow disseminating startup information on gatekeeper connect without having to create a new socket type.