MQTT-Explorer
MQTT-Explorer copied to clipboard
thomasnordquist
Improve handling of long messages
Hey
Long messages seem to get shortened. Example: https://gist.github.com/Wobbley/b71a19fe9e433247e5352a2c438d18c9
As you can see it simply cuts off the message. I checked and it displays fine in MQTTFX. Am I just looking in the wrong window? I used the copy value function from the window on the right.
OS: Ubuntu 18.04 LTS Version: 0.2.3
Currently, messages are truncated to 10kb.
- MQTT Explorer keeps a record of all topics/messages, I added the limit to prevent the app from running out of memory.
- Value preview would become unbearably slow if it would try to display a 5MB binary/text.
Silently truncating the messages is not good enough anymore, could be solved with a configuration option or at least a notification when a message has been truncated. What is your expected message size?
If you need an immediate solution have a look at: https://github.com/thomasnordquist/MQTT-Explorer/blob/50b2362a70b5a0dc106a0402ac1bbeece686768f/backend/src/index.ts#L47-L53
Change it and execute yarn build and yarn start
I assumed there was some performance related reasons for truncating the message. A note of some kind that the message has been truncated would be nice.
The message is big, and I understand the need for truncating it. More specifically it's two MP3 files (about 5 seconds long) and some more metadata. Is it possible to truncate it "per field" somehow? I don't mind if the huge base64 message is truncated, problem is that I don't get to see the rest of the content, hence confirm that the structure is correct.
MQTT Explorer works great for all of the other queues, so this might just be an edge case not worth pursuing.
The very first thing to do is notify the user of truncated content and add a (volatile?) option to increase the limit (10k / 100k / 1M / 5M).
Beyond ~1M / ~5M the preview and plot should truncate the content for performance reasons.
The Chrome Debugger handles long strings like this:
Long fields in JSON like this:
Redux Dev Tools trunk a little differently (infix truncation, keeping "...", {...}, [...] to indicate content type)
Destructuring the JSON and truncating large fields would be the coolest but I'd need to rewrite the JSON preview with an AST parser. I might have to do it either way for Plotting JSON values.
Notifying the user is great. An option to increase it is even better. Insomnia gives you the option to ignore the buffer, but gives you a clear cut warning that it will reduce performance, maybe even killing the application. It would be great not having to change application just to view those specific messages, but rather give it a go, and bear the consequences.
I would say that the Redux Dev Tools version is preferable, but I understand if that's a major re-write.
Except for the notification maybe, none of this will make it in the next release. I'm basically polishing up the current release, removing all the "paper-cuts" that have accumulated over time.
Preview: https://github.com/thomasnordquist/MQTT-Explorer/issues/87
Was caught out by this, initially thought I had a bug. Agree the minimum is to warn user the message has been truncated. Be great if we could increase the limit via settings as I do have to analyse larger messages from time to time which means switching tools :(
The limit needs to be better, but I'm not yet sure how.
Some ideas
- Static setting in the sidebar 1.1. Should this setting be permanent? (remembered through restarts) 1.2. Remember settings except "unlimited"
- Big warning in a cut off topic with the ability to increase the limit for this specific topic.
- Increase limit for the last x topics a user has selected (coupling between view state and the tree-model is pretty high)
- dynamic, based on current Ram usage (something like: <256MB unlimited, >256MB 100kb limit, > 512MB 10kblimit). This would be the most complicated approach, but most users would never see that there is a limit.
I was looking through my stale issue, and closing some of them. Should I just close this one as well? Still using your tool by the way, and got some co-workers to use it as well. Keep up the good work!
Hi @Wobbley ,
I'm interested in solving this one too, right now I see it's truncated at 20k.
Thank you
I just stumbled upon this recently when I was digging around zigbee2mqtt... (they put in one topic list of zigbee devices in network in JSON array of objects that have a lot of data - I have only 8 zigbee devices, and the topic with list have already ~21230 characters, I saw ppl having 20 and even more devices - how long they have this topic 😅 ).
Upvote for configurable value limits. @thomasnordquist - maybe a table or list in settings: where we can enter that exact topics of interest with their value max-length?
Hi Thomas, I.ve tried to extend mqtt explorer(installed under Windows 10 Enterprise) limit to 50KB , and I'm still trying to find in Application/Dev Tools in *.js files (under Page) but I didn't find yet MQTT-Explorer/backend/src/index.ts. Installed version is MQTT-Explorer 0.4.0-beta1. Can I kindly ask you to give more details to limit can be extended? Thanks a lot in advance.
I could need a solution to this as well. Not sure how to apply the 'solution' as well.
+1 for this! It doesn't look like this has been addressed as of early 2023 here. I'm working with payloads in the 50K to 100K range (zigbee network maps) and this is a real bummer of a limitation.
Hi all, I hope this email finds you well and safe. I was looking long time a way to write Thomas and now it seems I got this opportunity.
Related to this app, does exists the chances to be able to modify myself the limit of message up to 50K ? The app is installed on Win10 platform. I can copy the content of messages <= 20K, the 32K messages size is truncated.
Thanks and regards, Romeo
@.***
From: sean tearney @.*** Sent: Tuesday, March 14, 2023 10:38 AM To: thomasnordquist/MQTT-Explorer @.> Cc: Bucur, Romeo @.>; Comment @.***> Subject: Re: [thomasnordquist/MQTT-Explorer] Improve handling of long messages (#98)
Nu primiți des e-mail de la @.@.>. Aflați de ce este important acest lucruhttps://aka.ms/LearnAboutSenderIdentification
ATTENTION: This email was sent to you from an address outside of PMI. Please do not click on links or open attachments, unless you recognize the sender and know the content is safe. If you think this email is suspicious, please report it using the Phish Alert button. More info about it here https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpmicloud.sharepoint.com%2Fsites%2FRefDigitalWorkplace%2FSitePages%2FFAQ-Article-Others-Report-Phishing.aspx&data=05%7C01%7Cromeo.bucur%40pmi.com%7C4cd09621871548f2fa6b08db24676161%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C638143798662431819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XXLrtieRCF0dk0YqO5L7IaAV9Kz92I5ctmH7CVVZQ%2Bs%3D&reserved=0
+1 for this. At least add a warning that it's been truncated.
— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fthomasnordquist%2FMQTT-Explorer%2Fissues%2F98%23issuecomment-1467641738&data=05%7C01%7Cromeo.bucur%40pmi.com%7C4cd09621871548f2fa6b08db24676161%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C638143798662431819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEaSmd5%2FHpiJjZiLcE6GvN7oH%2BXFEb8b9EwffLYuUsM%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA2SSPP4T5T4AX34VYHZEVH3W4AU5NANCNFSM4HFOJ53Q&data=05%7C01%7Cromeo.bucur%40pmi.com%7C4cd09621871548f2fa6b08db24676161%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C638143798662431819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9PakG8wIrSp59da775MxZJWcikLGINmZhFwlWszSzKs%3D&reserved=0. You are receiving this because you commented.Message ID: @.@.>>
NOTICE: This communication may be confidential, legally privileged, proprietary in nature, or otherwise protected from disclosure. If you are not the intended recipient, or believe that you have received it in error, please contact the sender immediately and destroy the email and its attachments. To learn what information we collect and use about you, including monitoring communications, see our Privacy Notice at www.pmiprivacy.com/en/business-partner.
+1 for this! It doesn't look like this has been addressed as of early 2023 here. I'm working with payloads in the 50K to 100K range (zigbee network maps) and this is a real bummer of a limitation.
Yes, I do was working with Zigbee2MQTT too and for just ~10 devices I get payload about 226k (when z2m start it publish devices info+ config, and do it from time to time too).
At least information that payload is truncated would be sufficient.
+1
also ran into this today :( i think it would be great if we had some way to increase the max message size. thanks!
