eclipse
Mosquitto consumes much memory while "Saving in-memory database to /path/to/mosquitto.db"
Hi,
My mosquitto deployments has persistence true, and it runs in docker container. Normally the container uses ~13.4 GB memory, but when the scheduled "saving in-memory database to file" was performed, it eats all free memory (~5.5 GB) and caused out of memory and broken pipe in mosquitto logs, no matter autosave_interval was set to 300 (5 mins) or 3600 (1 hour). Actually I don't know if this is a normal behavior, a bug or something. I tried searching the issues but didn't see similar ones. So any information or suggestion would be appreciated. The docker image is "eclipse-mosquitto:2.0.14", and please let me know if you need any further information. Thanks a lot.
Are you saying that mosquitto normally has 13.4GB of RAM in use? That's a fair size. It shouldn't matter particularly for memory usage though, the largest allocation allowed should be on the order of 256MB, for an individual publish message - but freed immediately after. It is unusual, if you could provide some more details it could be helpful.
I have to say that it's the whole docker container which normally uses 13.4 GB of RAM, but since there's basically nothing else running inside the container, well, yes, I guess so.
I'd be glad to provide any info you might need for investigating this situation, but honestly I'm not very familiar with mosquitto so please guide me how to gather the information you need if possible. Or, let me know what information could be useful for investing, I can try to gather them.
Thank you.
Your mosquitto.conf (with sensitive bits removed) would be helpful, also any information on what messages are being sent and what client sessions there are. So do you have millions of retained messages, for example, and/or lots of persistent client sessions.
mosquitto.conf:
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
allow_anonymous true
listener 1883
protocol mqtt
autosave_interval 43200
$SYS/broker/#:
$SYS/broker/clients/total 374185
$SYS/broker/clients/inactive 366844
$SYS/broker/clients/disconnected 366844
$SYS/broker/clients/active 7341
$SYS/broker/clients/connected 7341
$SYS/broker/load/messages/received/1min 46623.40
$SYS/broker/load/messages/received/5min 46247.84
$SYS/broker/load/messages/received/15min 46136.16
$SYS/broker/load/messages/sent/1min 46678.96
$SYS/broker/load/messages/sent/5min 46269.62
$SYS/broker/load/messages/sent/15min 46144.37
$SYS/broker/load/publish/dropped/1min 5354.02
$SYS/broker/load/publish/dropped/5min 4935.50
$SYS/broker/load/publish/dropped/15min 4538.44
$SYS/broker/load/publish/received/1min 1214.76
$SYS/broker/load/publish/received/5min 1042.83
$SYS/broker/load/publish/received/15min 1062.64
$SYS/broker/load/publish/sent/1min 1537.75
$SYS/broker/load/publish/sent/5min 1330.13
$SYS/broker/load/publish/sent/15min 1331.27
$SYS/broker/load/bytes/received/1min 657458.44
$SYS/broker/load/bytes/received/5min 589963.20
$SYS/broker/load/bytes/received/15min 598127.52
$SYS/broker/load/bytes/sent/1min 637229.42
$SYS/broker/load/bytes/sent/5min 568529.69
$SYS/broker/load/bytes/sent/15min 576628.88
$SYS/broker/load/sockets/1min 352.66
$SYS/broker/load/sockets/5min 352.64
$SYS/broker/load/sockets/15min 346.43
$SYS/broker/load/connections/1min 209.77
$SYS/broker/load/connections/5min 206.17
$SYS/broker/load/connections/15min 201.27
$SYS/broker/messages/stored 2835320
$SYS/broker/messages/received 1344832771
$SYS/broker/messages/sent 1344397464
$SYS/broker/store/messages/count 2835320
$SYS/broker/store/messages/bytes 348971712
$SYS/broker/subscriptions/count 663101
$SYS/broker/retained messages/count 16122
$SYS/broker/publish/messages/dropped 593480020
$SYS/broker/publish/messages/received 60682835
$SYS/broker/publish/messages/sent 75278147
$SYS/broker/publish/bytes/received 16580573956
$SYS/broker/publish/bytes/sent 16442879922
$SYS/broker/bytes/received 28598189444
$SYS/broker/bytes/sent 24177476174
Thanks, that's helpful. Let's explain some of what you're seeing.
$SYS/broker/clients/connected 7341- the current number of connected clients - looks fine, if this is what you expect$SYS/broker/clients/disconnected 366844- the number of client sessions for persistent clients (withclean sessionset to false, or with asession-expiry-intervalof greater than zero. This is a large amount. I'm guessing you don't want this to be the case. I'm also guessing that the reason for this is that you're using clean session set false, but not setting a client id and your client library is generating a random one.$SYS/broker/load/messages/received/1min 46623.40~800 messages/s is fine - this includes all MQTT packets$SYS/broker/load/publish/dropped/1min 5354.02- this tells you how many messages are being dropped by the broker because a client has a full queue. This is a situation likely to occur with offline/disconnected persistent clients$SYS/broker/load/publish/received/1min 1214.76- 20 publish messages/s received is fine. It does mean that you have one publish per 40 other message - perhaps something to look into. If you had keepalive of 10 seconds your 7341 clients would be generating 730 PINGREQ messages per second, so perhaps that's ok$SYS/broker/load/sockets/1min 352.66- ~6 devices connecting to the network socket per second on average $SYS/broker/load/connections/1min 209.77 - 3.5 devices making a successful MQTT connection per second. What happened to the other 2.5 per second that opened a socket?$SYS/broker/messages/stored 2835320$SYS/broker/messages/received 1344832771- 1.3 billion messages received total$SYS/broker/store/messages/count 2835320- 2.8 million messages stored in the broker for some reason$SYS/broker/store/messages/bytes 348971712- 348MB of messages stored, or about 123 bytes per message$SYS/broker/retained messages/count 16122- this is just the retained message count, so we know that that vast majority of the stored messages are not retained messages and must be messages stored for offline clients$SYS/broker/subscriptions/count 663101- 600,000 subscriptions is roughly 2 subscriptions per client$SYS/broker/publish/messages/dropped 593480020600 million messages dropped, presumably for all of those offline clients
In conclusion, you've got a large number of offline clients that I don't believe should be there. They have around two subscriptions each and this results in a large number of messages being stored for those clients. This is what is causing the large amount of memory use in day to day operation.
I haven't tried to reproduce the case you're seeing so far, but I believe the above will be the cause of your problem. If you remove the current persistence file and start again you should be able to work again, and I'd then suggest looking into the logs to identify which clients are connecting with persistent sessions.
Hi ralight,
Thank you for your detailed explanation! So it looks like that my mosquitto deployment is working as expected according to the data I provided above, right? I'll try to understand your response further and confirm our use case with my colleagues to see if there's anything can be improved.
Another question I'm wondering is, how does the "saving in-memory database" feature work? Why does mosquitto consumes more memory while saving in-memory database to disk?
re the expired clients, try setting persistent_client_expiration to a suitable value for what you expect, it might just make them all "go away" on their own. Similar issue to https://github.com/eclipse-mosquitto/mosquitto/issues/3237
re the expired clients, try setting
persistent_client_expirationto a suitable value for what you expect, it might just make them all "go away" on their own. Similar issue to #3237
Thanks for the hint, I'll try the setting and monitor.
re the expired clients, try setting
persistent_client_expirationto a suitable value for what you expect, it might just make them all "go away" on their own. Similar issue to #3237
Hi @karlp , I tried setting persistent_client_expiration to 7d as a beginning to see if it helps. But the strange thing is, when it came to the time "saving in-memory database" was performed, the memory used by mosquitto container went up from 10.54 GB to 19.35 GB and did not fall as usual. Honestly I don't think this should be the expected behavior. According to the document, when persistent_client_expiration is not set, mosquitto does not expire persistent clients, so it looks to me that setting persistent_client_expiration should at lease expire some persistent and cause memory/space usage become lower. Please correct me if I'm misunderstanding. Any suggestion would be appreciated. Thank you.
Below is the content of $SYS/broker/# after setting persistent_client_expiration to 7d:
$SYS/broker/version mosquitto version 2.0.14
$SYS/broker/uptime 3035999 seconds
$SYS/broker/clients/total 375394
$SYS/broker/clients/inactive 367135
$SYS/broker/clients/disconnected 367135
$SYS/broker/clients/active 8259
$SYS/broker/clients/connected 8259
$SYS/broker/load/messages/received/1min 50200.91
$SYS/broker/load/messages/received/5min 50190.74
$SYS/broker/load/messages/received/15min 50097.72
$SYS/broker/load/messages/sent/1min 50213.43
$SYS/broker/load/messages/sent/5min 50196.99
$SYS/broker/load/messages/sent/15min 50100.38
$SYS/broker/load/publish/dropped/1min 3059.56
$SYS/broker/load/publish/dropped/5min 5146.31
$SYS/broker/load/publish/dropped/15min 5020.67
$SYS/broker/load/publish/received/1min 88.40
$SYS/broker/load/publish/received/5min 121.13
$SYS/broker/load/publish/received/15min 148.22
$SYS/broker/load/publish/sent/1min 478.68
$SYS/broker/load/publish/sent/5min 510.52
$SYS/broker/load/publish/sent/15min 554.01
$SYS/broker/load/bytes/received/1min 242465.46
$SYS/broker/load/bytes/received/5min 255955.36
$SYS/broker/load/bytes/received/15min 268668.00
$SYS/broker/load/bytes/sent/1min 215269.60
$SYS/broker/load/bytes/sent/5min 226501.93
$SYS/broker/load/bytes/sent/15min 237378.25
$SYS/broker/load/sockets/1min 468.00
$SYS/broker/load/sockets/5min 476.47
$SYS/broker/load/sockets/15min 490.13
$SYS/broker/load/connections/1min 279.29
$SYS/broker/load/connections/5min 285.78
$SYS/broker/load/connections/15min 297.56
$SYS/broker/messages/stored 2863551
$SYS/broker/messages/received 2318968881
$SYS/broker/messages/sent 2318484576
$SYS/broker/store/messages/count 2863551
$SYS/broker/store/messages/bytes 352288427
$SYS/broker/subscriptions/count 664228
$SYS/broker/retained messages/count 16550
$SYS/broker/publish/messages/dropped 703999867
$SYS/broker/publish/messages/received 69935582
$SYS/broker/publish/messages/sent 94416067
$SYS/broker/publish/bytes/received 20287952823
$SYS/broker/publish/bytes/sent 20881587844
$SYS/broker/bytes/received 37041217727
$SYS/broker/bytes/sent 31721779611
I remember experimenting with this config, but it was long ago and if memory serves, they will expire x amount of days after you set the config.
@Daedaluz Thanks for replying. Are you saying that if I set the config to 7d, for example, mosquitto will not start expiring clients until 7 days later? If this is the case, do you have any idea what if I change the config to a shorter period, like 3d? Will I have to wait for 3 days since the config was changed regardless how long I have had waited for before changing it again?
Yes that is what im saying. Though I cannot answer the second part. :/ Sorry
Though, I am quite sure it doesn't hurt to lower it now if you want. Either it starts purging the clients or it does not. Just set it to something reasonably low, so that you don't remove any clients that might still be in use but offline.
When I remember correctly the persistent_client_expiration config setting will not be applied for existing sessions already existing in you snashot file. It will only apply to new incoming sessions. As long as you never clear the existing sessions with the subscriptions from your broker it will never really shrink. You may try to use the db_dump tool to extract the clientid of all client and remove the unwanted ones by using a mosquitto_pub command line tool using the clientid you want to get rid of with the default clean session set to true. This way the sessions maybe remove from you broker.
