Mosquitto (2.0.18) broker crashes out of memory (OOM) when retaining messages

[vfpnyu]

Mosquitto 2.0.18 Linux VM Redhat 9.3 Azure 2cpu 8gb

We have an MQTT setup where we are publishing messages on around 500,000 topics, and we have noticed when we have set the retention flag, that the mosquitto broker memory fills up and eventually crashes.

If we don't publish with retention, we easily have 4gb+ of free RAM on our broker machine. By our calculations (500,000 * 100bytes messages = 50mb) of RAM would be needed to retain the messages published. But we see that the broker memory just keeps increasing until it crashes.

What's interesting is when we look at broker logs:

(red square indicates period where messages are published with retention)

$SYS/broker/store/messages/bytes The number of bytes currently held by message payloads in the message store. This includes retained messages and messages queued for durable clients. Here the bytes held in message store reaches 50mb and then stays at that level (makes sense as this is the size of the messages needed retained)

But when we look at $SYS/broker/heap/current size The current size of the heap memory in use by mosquitto. Note that this topic may be unavailable depending on compile time options.

The heap memory usage just keeps growing until it hits 12gb+ of usage (n.b the peaks and sudden falls are where the broker has crashed OOM and we had to restart it)

Once we turn off publishing messages with retention, all is OK again.

Does anyone have any idea why this is happening? Can we force mosquitto to clean up its memory allocation?

[Daedaluz]

Are there any subscribers?

[vfpnyu]

Are there any subscribers?

We usually have 5 subscribers yes.

They are connecting to the topics through a bridge

[ralight]

Could you please share your config file?

[vfpnyu]

Could you please share your config file?

Some stuff has been commented out where we have tried various fixes, e.g the persistence to disk to avoid memory filling up (didn't work)

# Config file for the Mosquitto broker.
#

# ================================================================================
# General Configuration
# ================================================================================

per_listener_settings false

# User Authentication
password_file [PW FILE PATH]
acl_file [ACL FILE PATH]

# Log settings
log_dest file [LOG FILE PATH]
log_dest topic

log_timestamp true
log_timestamp_format %Y-%m-%dT%H:%M:%S

# Publish the system information to the $SYS/ topic(s) every 10 seconds.
sys_interval 10

# Control the technical broker behavior
#max_inflight_messages 0
#max_queued_messages 0
#max_connections 50
queue_qos0_messages false

log_type error
log_type warning
# log_type notice
log_type information
#log_type debug
log_type subscribe
log_type unsubscribe

# Data retention and persistence settings
#persistence true
#persistence_location [PERSISTANCE FILE PATH]
# autosave every 1 min (60 seconds)
#autosave_interval 60


# ================================================================================
# Default Listner
# ================================================================================
listener 8883

tls_version tlsv1.2

allow_anonymous false

require_certificate true

cafile [CERT FILE PATH]
certfile [CERT FILE PATH]
keyfile [CERT FILE PATH]

# ==================================================================================
# Additional Listner (Only for testing)
# ==================================================================================
listener 1883

tls_version tlsv1.2

require_certificate false

#===================================================================================
# Topic bridging
#===================================================================================


connection local-org-a-bridge
address localhost:1883

topic TOPICX/60/XXX/# out 0 "" org-a/

connection local-org-b-bridge
address localhost:1883

topic TOPICX/60/xxx/# out 0 "" org-b/

[vfpnyu]

Wonder if anyone has any suggestions? We are stumped aside from trying to reduce the number of messages retained (what would be a good number?)

These are our broker logs if thats interesting... heap memory is terrifying :P

{'topic': '$SYS/broker/bytes/sent', 'payload': '779365535156'}
{'topic': '$SYS/broker/bytes/received', 'payload': '906259276594'}
{'topic': '$SYS/broker/publish/bytes/sent', 'payload': '467949732794'}
{'topic': '$SYS/broker/publish/bytes/received', 'payload': '556865847749'}
{'topic': '$SYS/broker/publish/messages/sent', 'payload': '4411598280'}
{'topic': '$SYS/broker/publish/messages/received', 'payload': '5463069809'}
{'topic': '$SYS/broker/publish/messages/dropped', 'payload': '2880296903'}
{'topic': '$SYS/broker/heap/maximum', 'payload': '14709324968'}
{'topic': '$SYS/broker/heap/current', 'payload': '14709324088'}
{'topic': '$SYS/broker/retained messages/count', 'payload': '130402'}
{'topic': '$SYS/broker/shared_subscriptions/count', 'payload': '0'}
{'topic': '$SYS/broker/subscriptions/count', 'payload': '2'}
{'topic': '$SYS/broker/store/messages/bytes', 'payload': '10613755'}
{'topic': '$SYS/broker/store/messages/count', 'payload': '130402'}
{'topic': '$SYS/broker/messages/sent', 'payload': '4411653108'}
{'topic': '$SYS/broker/messages/received', 'payload': '5463144994'}
{'topic': '$SYS/broker/messages/stored', 'payload': '130402'}
{'topic': '$SYS/broker/load/sockets/15min', 'payload': '12.41'}
{'topic': '$SYS/broker/load/sockets/5min', 'payload': '17.09'}
{'topic': '$SYS/broker/load/sockets/1min', 'payload': '38.56'}
{'topic': '$SYS/broker/load/bytes/sent/15min', 'payload': '8906982.63'}
{'topic': '$SYS/broker/load/bytes/sent/5min', 'payload': '928943.38'}
{'topic': '$SYS/broker/load/bytes/sent/1min', 'payload': '2168110.14'}
{'topic': '$SYS/broker/load/bytes/received/15min', 'payload': '9792189.09'}
{'topic': '$SYS/broker/load/bytes/received/5min', 'payload': '51731.60'}
{'topic': '$SYS/broker/load/bytes/received/1min', 'payload': '17476.31'}
{'topic': '$SYS/broker/load/publish/sent/15min', 'payload': '51211.31'}
{'topic': '$SYS/broker/load/publish/sent/5min', 'payload': '5921.87'}
{'topic': '$SYS/broker/load/publish/sent/1min', 'payload': '13840.74'}
{'topic': '$SYS/broker/load/publish/received/15min', 'payload': '59611.70'}
{'topic': '$SYS/broker/load/publish/received/5min', 'payload': '332.65'}
{'topic': '$SYS/broker/load/publish/received/1min', 'payload': '115.68'}
{'topic': '$SYS/broker/load/publish/dropped/15min', 'payload': '39609.21'}
{'topic': '$SYS/broker/load/publish/dropped/5min', 'payload': '10124.23'}
{'topic': '$SYS/broker/load/publish/dropped/1min', 'payload': '22126.35'}
{'topic': '$SYS/broker/load/messages/sent/15min', 'payload': '51215.88'}
{'topic': '$SYS/broker/load/messages/sent/5min', 'payload': '5924.45'}
{'topic': '$SYS/broker/load/messages/sent/1min', 'payload': '13845.30'}
{'topic': '$SYS/broker/load/messages/received/15min', 'payload': '59614.41'}
{'topic': '$SYS/broker/load/messages/received/5min', 'payload': '334.72'}
{'topic': '$SYS/broker/load/messages/received/1min', 'payload': '119.93'}
{'topic': '$SYS/broker/clients/expired', 'payload': '0'}
{'topic': '$SYS/broker/clients/connected', 'payload': '5'}
{'topic': '$SYS/broker/clients/active', 'payload': '5'}
{'topic': '$SYS/broker/clients/disconnected', 'payload': '0'}
{'topic': '$SYS/broker/clients/inactive', 'payload': '0'}
{'topic': '$SYS/broker/clients/maximum', 'payload': '15'}
{'topic': '$SYS/broker/clients/total', 'payload': '5'}
{'topic': '$SYS/broker/uptime', 'payload': '249207 seconds'}
{'topic': '$SYS/broker/version', 'payload': 'mosquitto version 2.0.18'}

[karlp]

I'd consider turning on persistent_client_expiration and restarting, I'd suspect you're storing far more than you think for clients that may not actually be there any more, and arne't going to come back.

[vfpnyu]

If I look at our clients, we only have 5 and none are inactive. So i'm not sure if this will fix anything. I agree its a similar issue to #3273

Our stored/retained message counts are also similar, so i think all messages being stored are retained messages, which is different to 3273

[vfpnyu]

I'm interested in other's configurations while retaining messages. We notice if we reduce the number of topics that we retain messages on, the memory still increases steadily but at a slower rate, even as the amount of retained messages is stable.

How many topics/retained messages do other users store and how much memory does that consume?