Raft-based clustering support for Mosquitto

[Elia-Renzoni]

Hi @ralight

I am exploring the idea of adding a Raft-based consensus mechanism to Mosquitto, to provide clustered/distributed broker functionality.

My questions are:

• Would you be open to such a contribution being developed directly within the Mosquitto project?

• If not, since Mosquitto is dual-licensed under the EPL-2.0 and the EDL-1.0, would it be acceptable for me to create a fork that integrates Raft, and possibly redistribute it under one of the available licenses?

From my understanding, both licenses would allow this (EPL with weak copyleft obligations, EDL in a more permissive way), but I’d like to confirm with you.

Thanks for your feedback.

[ralight]

Hi, thanks for the interest! First off, you're absolutely allowed to make forks - as you've noted mosquitto is dual licensed under EPL-2.0 and the EDL-1.0. As long as you comply with the terms of the license and redistribute it under that license.

There are no plans to integrate clustering into the mosquitto core. The aim of the project is to stay as lightweight and simple as possible. Clustering is definitely an advanced and complicated feature that I believe goes beyond that scope and would make the broker harder to use in general.

For the sake of transparency, I have to also say that Cedalo, who I work for, provide a pro version of mosquitto which includes a clustered version based on the raft protocol. This represents a conflict of interest for me as the project lead, however exactly based on the experience of working on that project I can say it does complicate the broker beyond what I think is appropriate here.

[anlexN]

@ralight if no cluster, how to implement millions of connections in the mosquitto brokers?

[ralight]

@anlexN Depending on what you need, this is completely possible with the broker as it is. In my experience the typical use case where there are a large number of connections is many sensors publishing to a broker and a few subscribers consuming the messages and acting on them (N publishers to 1 subscriber). In that case, having an array of independent single brokers works perfectly fine, there is no need for a cluster. You can also support communication back to the publishers with a little effort on the application side to ensure all brokers receive the message.

If you want the more unusual case of 1 publisher to N subscribers, then you could do this with a single broker bridging to multiple brokers, each of the multiple brokers having many subscribers or a third layer of brokers for more fan out if required.

If you want M publishers talking to N subscribers then you need a single broker or a cluster, so would ultimately be limited on a single node.

If you're really looking at millions of connections then I'd expect you have a commercial requirement for that and would benefit from commercial support. Perhaps you would like to talk to us at https://cedalo.com/

[anlexN]

@ralight Thank you for your reply, you are my friend!

What email are yours? I want to add you.