Documentation: What parts of the zmq library does libzmtp support?

[vitiral]

This library looks amazing and exactly what I am looking for?

• Does this library support pub / sub?
• Does this library require threading?
• Does this library require a memory manager?

In general, what features are supported and what does the library require to operate?

Thanks a ton!

[vitiral]

From looking at the source code, it seems that it supports the ZMQ_DEALER socket and something called CHANNEL, which I don't see in the zmq socket documentation

How difficult would it be to implement PUB/SUB or XPUB/XSUB?

[vitiral]

Delving deeper into the zmq documentation, it looks like one could build majordomo on top of this library, but pub/sub is completely different. Is that about right?

[hintjens]

The libzmtp is a minimal library meant for clients only, at this stage. That means no multithreading, no asynch I/O, and support for the DEALER socket which is the main client socket. (We'd eventually replace with support for the CLIENT socket when/if that becomes mainstream.)

It should be relatively easy to add SUB socket support. PULL and PUSH are equivalent to DEALER (with half the API disabled each time).

On Fri, Apr 3, 2015 at 5:51 PM, Garrett Berg [email protected] wrote:

Delving deeper into the zmq documentation, it looks like one could build majordomo on top of this library, but pub/sub is completely different. Is that about right?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89334935.

[vitiral]

What about a memory manager? Do you use malloced data that you later free? Does the system have to have ways to deal with defragmentation?

[hintjens]

Yes, it uses the heap. You can see this quite simply in the code.

On Fri, Apr 3, 2015 at 9:47 PM, Garrett Berg [email protected] wrote:

What about a memory manager? Do you use malloced data that you later free? Does the system have to have ways to deal with defragmentation?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89405244.

[vitiral]

The reason I ask is because I have written a memory manager for microcontrollers, tinymem. It is currently in Alpha state (usable but needs more tests and features), but it would allow microcontrollers to use you library.

I have an overall vision:

• Microcontrollers that can use pub/sub to provide easy interfacing with larger systems
• ZMQ implemented on a generic socket structure so that any communication protocol can be used:
  • in particular I want to communicate over TCP/IP, uIP, Zigbee and CAN Bus.

I would like to make a branch of libzmtp to use tinymem (as soon as tinymem is in Beta). The branch would be written in such a way that a developer could disable tinymem if they were on a system that supported malloc/free.

I would also like to extend libzmpt to support pub/sub.

Would these things interest you? I will write the code and do a pull request to use tinymem.

[hintjens]

Sounds fun, definitely worth exploring.

On Fri, Apr 3, 2015 at 10:26 PM, Garrett Berg [email protected] wrote:

The reason I ask is because I have written a memory manager for microcontrollers, tinymem https://github.com/cloudformdesign/tinymem. It is currently in Alpha state (usable but needs more tests and features), but it would allow microcontrollers to use you library.

I have an overall vision:

• Microcontrollers that can use pub/sub to provide easy interfacing with larger systems
• ZMQ implemented on a generic socket structure so that any communication protocol can be used:
  • in particular I want to communicate over TCP/IP, uIP, Zigbee and CAN Bus.

I would like to make a branch of libzmtp as soon as tinymem is in Beta that uses tinymem. The branch would be written in such a way that a developer could disable tinymem if they were on a system that supported malloc/free.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-89410899.

[vitiral]

I've looked over how zmq actually does it's filtering, and I'm worried that it is not a protocol that can be realistically ported to microcontrollers.

I've wondered for a while on how they do their matching, and it looks like they use an inverted bitmap to search through a set of strings and determine matches.

I don't think this is an embeddable algorithm. I've actually begun work on a protocol with a much simpler matching scheme that I call TMQ. This uses tokens instead of strings for matching, which means I can use just hash table lookups for everything.

Thoughts?

[hintjens]

That whitepaper describes the matching we used in OpenAMQ, over a decade ago. It's very powerful, and absolutely not for embedded systems.

ZeroMQ uses a very different matching model, see http://rfc.zeromq.org/spec:29. Matching is done by the publisher.

So while TMQ looks fine, it's based on a rather wrong assumption of what ZeroMQ does.

On Sun, Apr 26, 2015 at 9:31 PM, Garrett Berg [email protected] wrote:

I've looked over how zmq actually does it's filtering, and I'm worried that it is not a protocol that can be realistically ported to microcontrollers.

I've wondered for a while on how they do their matching, and it looks like they use an inverted bitmap http://zeromq.org/whitepapers:message-matching to search through a set of strings and determine matches.

I don't think this is an embeddable algorithm. I've actually begun work on a protocol with a much simpler matching scheme that I call TMQ https://github.com/cloudformdesign/tmq. This uses tokens instead of strings for matching, which means I can use just hash table lookups for everything.

Thoughts?

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-96423118.

[vitiral]

That's good to hear. So how much memory do you think a publisher would take under various conditions? Do you really think it is doable on a microcontroller?

I've just had a pretty big breakthrough on my memory manager that will require a bit of an overhaul. I'll let you know when I'm closer :)

[hintjens]

A simple publisher needs no significant memory at all. An embedded device just needs to broadcast its messages; subscribers can filter them if they want.

On Mon, Apr 27, 2015 at 12:14 AM, Garrett Berg [email protected] wrote:

That's good to hear. So how much memory do you think a publisher would take under various conditions? Do you really think it is doable on a microcontroller?

I've just had a pretty big breakthrough on my memory manager that will require a bit of an overhaul. I'll let you know when I'm closer :)

— Reply to this email directly or view it on GitHub https://github.com/zeromq/libzmtp/issues/40#issuecomment-96439879.