rabbitmq-server
rabbitmq-server copied to clipboard
rabbitmq
Retained messages not sent to wildcard subscribers
Using the RabbitMQ Alpine docker image, which contains: RabbitMQ 3.6.14 on Erlang 19.3
Custom Docker image created to add the:
RUN rabbitmq-plugins enable --offline rabbitmq_mqtt
EXPOSE 1883
Commands to enable MQTT.
Docker started with:
docker run -d --hostname rabbit --name rabbit -p 8080:15672 -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbitmq_mqtt.retained_message_store [mqtt.retained_message_store [rabbit_mqtt_retained_msg_store_dets] mqtt.retained_message_store_dets_sync_interval [2000]]" rabbit-mqtt
to enable message retention to disk.
There is what appears to be a bug in that retained messages are not sent to a receiver which starts after the retained message is set and which uses a wildcard to subscribe.
Simple send code:
import paho.mqtt.client as mqtt
import time
broker_address="172.17.0.2"
client = mqtt.Client("sender", False, mqtt.MQTTv311)
client.connect(broker_address)
client.publish("/examples/test1", "test 1 Retained", retain=True)
client.loop()
Simple receive code:
import paho.mqtt.client as mqtt
import time
def on_message(client, userdata, message):
print("received topic %s QoS %d RF %d received message %s" % (message.topic, message.qos, message.retain, str(message.payload.decode("utf-8"))))
broker_address="172.17.0.2"
client = mqtt.Client("listener", False)
client.on_message=on_message
client.connect(broker_address)
client.subscribe("/examples/test1", 1) <-- This works
client.subscribe("/examples/#", 1) <-- This does not
client.loop_forever()
To reproduce the issue:
Comment out one of the two subscribes.
Start receiver python receive.py
Run sender python send.py
Messages will be sent and received as expected. Exit receiver and restart it.
If the subscription is to the specific topic, the retained message is received. If the subscription is to the wildcard topic, the retained message is not received.
Using VerneMQ, the retained message is always received using either subscription form.
Retained messages have at least one known limitation: the store is node-local (that is documented). I can't think of a reason for wildcards to not have an effect with just one node, thank you for providing steps to reproduce.
Is this going to be fixed? I am having the same issue and just discovered this thread.
@mdford it's not a priority but this is open source software, so those who need this badly are free to look into supporting wildcards (it's not entirely trivial, at least with the current plugin architecture).
Hi, Somebody know how solve this....I migrate all my plattform from mosquitto to Rabbitmq, but I dont know that Rabbit has this limitation..
For now I has to suscribe to each topic when I need recibe retained message, but maybe there are something to configure for this works... any idea?
regards
Hi,
I'm not an Erlang developer, I couldn't implement a hello world, but from what I understand the handle_call for fetch should return a list of replies instead of a single reply and instead of ets:lookup, ets:match (or similar) should be called with a search query generated from the topic. However in case of many retained messages, the performance impact would be huge so I guess it should be a third retained message store. Perhaps some fancy indexing could be implemented.
https://github.com/rabbitmq/rabbitmq-mqtt/blob/b19eeb4edf1003ca2fb39884319a4755cda88a09/src/rabbit_mqtt_retainer.erl#L85-L91
https://github.com/rabbitmq/rabbitmq-mqtt/blob/d367b61e4c83121a41b8ec3f5131488c3de6bded/src/rabbit_mqtt_retained_msg_store_ets.erl#L52-L54
Learning Erlang is on my TODO list but I'm afraid I will not be able to send a pull-request anytime soon.
A sequential scan over all retained messages would not be acceptable. RabbitMQ already has a trie-based topic routing implementation (one, two). It has to be adapted.
On top of this, retained message store really should be distributed, which is somewhat similar to what we did in rabbitmq/rabbitmq-mqtt#91 for duplicate client ID tracking. It is unlikely that we will delay 3.8 because of this issue.
It has not been. Unless someone would be interested in contributing (this is a fairly non-trivial task, although much smaller if limited to an in-memory implementation), this is unlikely to be addressed soon because there are much higher priority items for the core team, even in this plugin alone.
This seemingly non-compliant behavior is quite troubling as an end-user :( especially since it’s known and been floating around for 2.5 years..
can't figure how its not supported yet
Comments like the above do not help the discussion. This is the reason why this feature is not yet supported:
there are much higher priority items for the core team, even in this plugin alone
@sobelman and everyone else who has commented here, please take note of the "Sponsors" section of this feature added to RabbitMQ:
https://github.com/rabbitmq/rabbitmq-server/pull/2654
Hi everyone.
We recently switched to RabbitMQ from MQTT for professional reasons, and were disappointed to see this very troubling issue. We use MQTT for our legacy applications and most of them use wildcard-based subscriptions for various reasons, and it is not really possible for us to do the switch to fully qualified topic names in most of those applications.
Returning to Mosquitto for MQTT could be an option if Mosquitto didn't fail us due to the high volume of messages.
We would like to know what it would take for this issue to be resolved as quickly as possible without disrupting your schedule.
Thank you very much!
We would like to know what it would take for this issue to be resolved as quickly as possible without disrupting your schedule.
Pay VMware or another vendor to implement the feature. Erlang Solutions has provided excellent work for paying customers.
https://www.rabbitmq.com/#support
Is this issue actually still present in the current version of RabbitMQ? If yes, is it somewhere on the roadmap for the near future to get fixed?
It is. It's not the short term roadmap. Making the MQTT plugin significantly more efficient is, that's what the majority of users care about.
Sad to see this is still an issue. Took me all night to figure out it was a limitation and I did not do anything wrong. I understand the decisions the dev. team have to make. For me this means looking for another solution as for my personal projects this is a must have.
I've noticed it seems this does not only relate to retained messages. Also QoS 0 and 1 in a clustered rabbitmq setup seem to not be able to subscribe to wildcards.
When I subscribe to specific topics, it does work.
@robbe-haesendonck Please use GitHub Discussions for questions. This issue is about a specific change in the MQTT plugin.
RabbitMQ MQTT plugin does support wildcards and always has, even though due to an internally used topic structure separator that's different from that in MQTT, there is a couple of edge cases.
We'd need an executable way to reproduce what you see (using Mosquitto CLI tools or a piece of code)>