rabbitmq-server icon indicating copy to clipboard operation
rabbitmq-server copied to clipboard

Support wildcards in MQTT topic filters matching retained messages

Open ansd opened this issue 2 years ago • 8 comments

MQTT spec:

the Server MUST send the retained messages matching the Topic Filter of the subscription to the Client

However, RabbitMQ does not send retained messages for Topic Filters containing wildcards.

The solution should comply with at least the following requirements:

  1. The solution should be (somewhat) efficient. For example if there are 100k retained messages, if a new client subscribes, the solution must not perform a full table scan over all 100k retained messages/topics.
  2. The solution must not overload the broker. For example if there are 100k retained messages, and a client subscribes with a topic filter matching 50k of these retained messages, the broker must not send all 50k messages in one go.

ansd avatar Jul 10 '23 13:07 ansd

A future RabbitMQ release will replicate retained messages in the cluster and also send retained messages matching topic filters containing wildcards.

Does this issue also takes track also of the first part about the retained messages that are not replicated in the cluster? In my case this is a big limitation, is there any work in progress or ETA for that?

robertsLando avatar Feb 28 '24 14:02 robertsLando

Does this issue also takes track also of the first part about the retained messages that are not replicated in the cluster?

No, this is tracked in https://github.com/rabbitmq/rabbitmq-server/issues/8096

is there any work in progress or ETA for that?

No. For the time being you can write your own plugin as for example done in https://github.com/gery0815/rabbitmq-mqtt-retained-msg-redis

ansd avatar Feb 28 '24 14:02 ansd

@ansd Thanks for that! Is there any mongodb alternative? Otherwise we could create one

cc @getlarge

robertsLando avatar Feb 28 '24 14:02 robertsLando

Is there any mongodb alternative?

I'm not aware of a mongodb plugin.

ansd avatar Feb 28 '24 15:02 ansd

Hi, we ran into the same issue. Even though rabbitmq is our preferred broker, we can't use it because of this issue. We unfortunately we don't have anybody knowing anything about writing erlang. Thus we can't provide a PR for this.

We are willing to financially support anybody, who is willing to fix this. Who would be the right point of contact? @ansd?

juergen-albert avatar Jul 09 '24 15:07 juergen-albert

@juergen-albert are you referring to only this issue or also to https://github.com/rabbitmq/rabbitmq-server/issues/8096? If you are referring to only this issue, I assume you run only a single node RabbitMQ cluster?

ansd avatar Jul 09 '24 16:07 ansd

Right now, I refer only to this issue and yes we are currently run on a single node.

juergen-albert avatar Jul 10 '24 12:07 juergen-albert

@juergen-albert please contact [email protected] to become a commercial VMware Tanzu RabbitMQ customer in order to discuss prioritisation of this feature.

ansd avatar Jul 10 '24 13:07 ansd