emqx
Exporting data can lead to instability and crashing the underlying OS due to memory starvation.
What did I do
Used export data to save a large number of MQTT messages as JSON.
What happened
It appears MQTTX extracts all the messages out of the database file (.config/MQTTX/MQTTX.db) and constructs a large in memory document that is then saved to disk of the users choice via a file dialog. If there are a large number of messages the in memory document uses all available memory and leads to an unstable or unresponsive system or OS crash. The current version of MQTTX will crash a system with 32GB of RAM when exporting a large dataset.
Expected
Ask the user for the format of the export, the file location and then stream the contents of the database in the correct format to the file at a record/message level without using an egregious amount of memory.
Environment
- OS: OS version, Linux 5.4.0-58-generic
- MQTTX version, V1.11.0
More detail
More detail include log information or screenshot
This is indeed an issue that needs optimization. We'll improve the export functionality to use a streaming approach instead of loading everything into memory. Thank you for reporting this problem.
Welcome to try https://github.com/emqx/MQTTX/releases/tag/v1.12.1, where this issue has been addressed.