Delay in High Performance scenario

[felixmondelo]

Describe your question

We have an IoT scenario where we use the MQTTnet clinet to publish 12.000 msg/sec in different topics.

Our scenario is working fine, but message are arriving with an increasing delay of several minutes to the MQTT broker.

foreach (var mqttMessage in mqttMessages)
{
   totalMessages += 1;
                
   MqttClientPublishResult publishResult = await mqttClient.PublishAsync(mqttMessage, CancellationToken.None);

   if (publishResult.ReasonCode != MqttClientPublishReasonCode.Success)
      _logger.LogError($"[{MODULE_NAME}] Error publishing. Reason Code {publishResult.ReasonCode} {publishResult.ReasonString}");
}

After the await seems that MQTTnet continues working behind the scenes publishing the messages, because if we stop publishing messages ... the broker continues receiving for a while.

How I can manage this delay? We need that messages arrive to the broker in near-realtime.

Which project is your question related to?

• Client

[MD-V]

Hello @felixmondelo,

are you using ManagedMqttClient?

[felixmondelo]

Hello @MD-V,

First I tested with normal MqttClient, now I change to ManagedMqttClient and the result is the same ... 20 minutes delayed:

foreach (var mqttMessage in mqttMessages)
{
   totalMessages += 1;

   await mqttClient.EnqueueAsync(mqttMessage);                
}

// Wait until the queue is fully processed.
SpinWait.SpinUntil(() => mqttClient.PendingApplicationMessagesCount == 0, 10000);

[felixmondelo]

If I don't wait until the queue is fully processed, the queue grows continuosly.

This means that the client library can't process 12k messages per second due to something (CPU, Memory, ...)?

Can I modify something in my code to improve the performance?

[chkr1011]

The most important thing is to await the publishes. Otherwise there will be only lots of tasks generated which will run in the future etc.

When publishing the packet gets serialized and handed over to the .net TCP socket. So it is written to it. If there is any delay it must come from there.

For high performance scenarios please use the regular client only. It has the best performance of all client types.

Which version of the library do you use?

[logicaloud]

There is a RoundtripBenchmark included in the sources at MQTTnet\Source\MQTTnet.Benchmarks that processes 100000 messages locally (local client, local broker; option b in the MQTTnet.Benchmarks.exe). On my lowly 9 year old test machine (Intel Haswell), the latest MQTTnet version delivers 100000 messages in about 2 seconds. For 12000 messages it takes proportionally less time (~200 ms).

@felixmondelo, besides info on the MQTTnet version in use, could you provide some details regarding the broker and network setup?

[chkr1011]

Please check the samples and if they behave in the same way. Without a full code snipped we can only guess what the issue might be. Please also remove the SpinWait.SpinUntil(() => mqttClient.PendingApplicationMessagesCount == 0, 10000); from your code. Obtaining the value will require several locks and countings and since SpinWait does this very very often you might already get performance issues from there.

As @logicaloud mentioned also please execute the tests from the Benchmark project and see if they also have huge delay.

If the issue persists please reopen this ticket.