meshtastic
[Bug]: json (payload) utf-8 characters converted to \n0000
Category
Other
Hardware
Heltec V3
Firmware Version
at least 2.2.4 and above
Description
When sending a message from my Mestastic node nothing seems to be wrong, the thing that does goes wrong is this. When using a MQTT Broker. As soon as utf-8 characters are used, their converted into \u0000 no matter what letter is beïng used.
Messages do arrive the correct way on other devices, but when using mosquitto_sub or any other sort of script to output the MQTT json data, it's shown as garbage. Example:
mosquitto_pub -h localhost -u {user} -P {pass} -t 'msh/3300-3329/2/json/mqtt/!{1}' -m '{"channel": 0,"from":{2},"type":"sendtext","payload":"äáàéëèóöò etc. etc."}'
Which output:
{"channel":0,"from":{1},"hops_away":0,"id":{2},"payload":{"text":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000 etc. etc."},"sender":"!da5c87d4","timestamp":1712428198,"to":{3},"type":"text"}
The sent data via mosquitto_pub is correctly shown on other devices, however... exporting the data from the MQTT broker shows those \u0000's.
Looks as if Mashtastic is converting all json data into unicode (even utf-8 characters) messing the output data up.
Relevant log output
{"text":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000 etc. etc."},"sender":"!da5c87d4","timestamp":1712428198,"to":{3},"type":"text"}
@GerritS76 You have ruined my dreams :) opening it again I am waiting for ages for this fix because in my language it is a lot of ąęść 😃
I can confirm this issue also in firmware 2.3.13.83f5ba0 Beta, look like the MQTT JSON text not decode with UTF-8.
Can confirm this issue still exists.
My guess is that something in this function is not quite right -- https://github.com/meshtastic/firmware/blob/master/src/mqtt/JSONValue.cpp#L833
Specifically these lines -- https://github.com/meshtastic/firmware/blob/master/src/mqtt/JSONValue.cpp#L854-L863
I have this problem too, I'm patiently waiting for this to be taken into account.
The JSON library being used, and the JSON on device in general is limited and does not support special characters. You need to use protobufs and parse json externally for full functionality.
This issue has been mentioned on Meshtastic. There might be relevant details there:
https://meshtastic.discourse.group/t/sending-json-text-via-mqtt/12030/9
