Callback `on_message` called multiple times when using MQTT v5.0

[anik01ic]

Hello,

why is the on_message function called 3 times when I am using MQTT v5.0, and exactly once when I am not using MQTT v5.0? The mosquitto version is 2.0.14. Here is the code:

#include <mosquitto.h>
#include <mqtt_protocol.h>
#include <thread>
#include <cstring>

#define SERVER_HOST "127.0.0.1"
#define SERVER_PORT 1883

/* Callback called when the client receives a message. */
static void on_message(struct mosquitto* mosq, void* obj, const struct mosquitto_message* msg, const mosquitto_property* props)
{
    printf("%s called with topic %s\n", __FUNCTION__, msg->topic);
    return;
}

int main(void)
{
    struct mosquitto* mosq_;
    char msg[] = "testmsg";
    int qos = 0;

    mosquitto_lib_init();
    mosq_ = mosquitto_new("tx", false, NULL);
    if (mosq_ == NULL)
        return 1;

    // Commenting this function call results in the `on_message` being called exactly once.
    // With this line it is called 3 times (for each mosquitto_subscribe).
    mosquitto_int_option(mosq_, MOSQ_OPT_PROTOCOL_VERSION, MQTT_PROTOCOL_V5);

    mosquitto_message_v5_callback_set(mosq_, on_message);
    mosquitto_connect(mosq_, SERVER_HOST, SERVER_PORT, 60);
    mosquitto_loop_start(mosq_);
    mosquitto_subscribe(mosq_, NULL, "#", qos);
    mosquitto_subscribe(mosq_, NULL, "1/#", qos);
    mosquitto_subscribe(mosq_, NULL, "1/2/#", qos);

    mosquitto_publish_v5(mosq_, NULL, "1/2/3", strlen(msg), static_cast<void*>(msg), qos, false, NULL);
    std::this_thread::sleep_for(std::chrono::seconds(1));

    mosquitto_disconnect(mosq_);
    mosquitto_destroy(mosq_);
    mosquitto_lib_cleanup();

    return 0;
}

Regards

[karlp]

This is related to how mqttv5 worded the requirements around overlapping subscriptions, and has been like this since https://mosquitto.org/blog/2019/06/version-1-6-3-released/ (see "Fix MQTT v5 overlapping subscription behaviour")

See also: section 3.3.5 here: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html and section 3.3.4 here http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html

I suspect that, even though you aren't specified a subscription ID, the fact that if they are used in v5, they must be used, has resulted in mosquitto choosing to send one for each, rather than a single one, with a chain of subids.

see also https://mosquitto.org/man/mosquitto-conf-5.html under "allow_duplicate_messages"

[anik01ic]

Okay, I understand now, thanks. But is there a way to tell the broker to send only one message (as it was with MQTT v3)? I have set the allow_duplicate_messages to false, but still getting multiple messages. Maybe it is impossible with MQTT v5?

[sgoll]

For reference, a question related to this issue was asked in #944 back in 2018—regarding MQTT 3 [MQTT-3.3.5-1] but the same applies to the specification of MQTT 5 [MQTT-3.3.4-2].

[ralight]

Hello,

My perspective on this is that I absolutely do not like having to make checks for duplicate messages. If it were entirely my choice, I'd completely remove that option so that duplicate messages occur for overlapping subscriptions. Preventing duplicate messages means either storing a list of client ids with the message to show where it has been sent, or storing a list of messages that a client has had sent to it, plus being able to efficiently search in those lists. The means at the absolute minimum there need to be as least as many memory allocations as there are per subscribing clients, and that for every single message that comes in. If you had say 1000 clients and 10,000 msgs/s then that's 10M allocations and frees that could be avoided per second.

Now, in principle allow_duplicate_messages could be changed to apply to MQTT v5 as well. As I've suggested, I'm against that idea. I'm open to being persuaded though - does this situation really exist in practice? Is it absolutely the case that the client couldn't just subscribe to the single topic and hence make this point moot?