emqx
When the data volume exceeds 7000, mqttx will become abnormally stuck
在数据量超过7000的时候，mqttx会变得异常卡顿，且内存占用超过3000M。我看了一些代码，我觉得原因大概是数据库操作都是同步的 await，ui那边好像也是同步的，造成ui会等着数据库，就会卡死。所以也许改进方案是前端这边异步获取数据？或者数据获取需要分页？ When the data volume exceeds 7000, mqttx will become abnormally stuck, and the memory occupation exceeds 3000M. I read some codes. I think the reason is that the database operations are synchronized await, and the UI seems to be synchronized. As a result, the UI will wait for the database and get stuck. So maybe the improved solution is to get data asynchronously on the front end? Or data acquisition require paging? 但是内存占用这个我也不清楚，数据量有这么大吗？还是因为electron的缘故，所以占用内存很大？不过只要数据量不那么大的话，内存占用也不是很大，卡顿会发生在连接源切换、点击连接 等等时候 But I don't know the memory occupation. Is there such a large amount of data? Or does it occupy a lot of memory because of electron? However, as long as the amount of data is not so large, the memory occupation is not very large. Froze will occur when the connection source is switched, the connection is clicked, and so on
看代码时候的一些log，仅供参考 Some logs when watching the code， for reference 文档：mqttx 项目文档.note 链接：http://note.youdao.com/noteshare?id=cb56c214ed05fc990f47fd2cece28ba2&sub=5A063C90D8C14321899C1CAE0C111DE5
@starplatinum3 Sorry for my late reply. YES, your analysis and research are very careful, thank you very much as a user can help us so much. In fact, in addition to that, we do a lot of additional message copying when the message arrives, so he will always have a lot of memory occupied.
But we are in the process of refactoring the underlying code architecture to address these issues, and @oceanlvr can provide you with more information about this, or perhaps you have suggestions, which we can discuss here.
I'm glad you can pay attention to MQTT X and do so much analysis, I read your notes, in fact, most of the problems you mentioned come from the irregular behavior of our early development, so in the next version of MQTT X problems and Focus on data optimization, memory usage, etc. 你好，很高兴您能关注 MQTT X 并做了这么多分析，我阅读了您的笔记，实际上您说的问题中大部分来自于早期我们开发的不规范行为，因此在下个版本的MQTT X将会着重关注数据流、内存占用和SQL优化等问题。
For the memory problem, the main reason is that we use a lot of data copies, there are multiple copies of data in service (typeorm in Node environment), UI (component's data/vuex state), and there are a lot of repeated full read and write operations, These will be the focus of our optimization later, and the current plan is to do a little simple monitoring and tracking. 对于内存问题，现在主要原因是我们用了大量的数据拷贝，在service（Node 环境下的 typeorm）、UI（组件的data/vuex state）有多份数据，而且有大量重复的全量读写操作，这些将是我们后面的优化重点，目前计划是要做一点简单的监控和埋点。
For the performance problem, the main reason now is the problem of full message reading (you have also analyzed, this part of the performance is obviously degraded (there is a big freeze when clicking on the connection, we plan to completely reconstruct the data flow, reconstruct the The plan is temporarily put on hold, sorry to bring you a bad UI experience) 对于性能问题，现在的主要原因是 message 全量读入问题（您也分析了，这部分性能降低的很明显（在点击连接的时候有很大的卡顿，我们打算完全重构数据流，重构计划暂时搁置了，很抱歉给你带来不好的UI体验）
For asynchronous acquisition of data, MQTT X acquires data asynchronously, but the dependencies between services are not thoroughly analyzed, and some paths will have unnecessary synchronization. In this part, I want test cases to ensure the speed. 对于异步获取数据，MQTT X 都是异步的获取数据，只不过业务之间的依赖没有做到分析透彻，有些路径会有不必要的同步，这部分我想让测试用例去保证速度。
We have also considered all the issues you mentioned. Seeing that you are very enthusiastic about MQTT X, if you are interested, we can build the next big version of MQTT X together. Not only the above three points, but also many interesting things We can add in, such as e2e testing\unit testing\plugin\scripting system\new stream data flow\faster data query, SQL monitoring, etc. @starplatinum3 您提到的所有问题我们也有些许考虑，看到您对 MQTT X 抱有很高的热情，如果有兴趣的话可以一起构建下一个大版本的 MQTT X，不光是上面三点，有很多有趣的东西我们可以加进去，比如 e2e 测试\单元测试\插件化\脚本系统\新的stream数据流\更快的数据查询、SQL监控等 @starplatinum3
如果能帮上忙的话，就非常荣幸了，只是我现在水平不足，希望我通过学习之后能达到一定水平，提出一定的解决方案吧 It would be a great honor if I could help, but I'm not good enough now. I hope I can reach a certain level and put forward some solutions after learning
Apologies for the long wait, welcome to https://github.com/emqx/MQTTX/releases/tag/v1.9.1, that should solve the problem and I will close this issue.