Broken JSON when sending payload larger than 127 bytes

[pruwait]

My PubSubClient send to mosquitto broker topic Light/relay/json_status with payload: {"r1":1,"r2":1,"r3":1,"r4":1,"r5":1,"r6":1,"r7":1,"r8":0,"r9":1,"r10":0,"r11":0,"r12":0,"r13":0,"r14":0,"r15":0,"r16":0,"r17":0,"r18":0,"r19":0,"r20":0,"r21":0,"r22":0,"r23":0,"r24":0,"r25":0,"r26":0,"r27":0,"r28":0,"r29":0,"r30":1,"r31":0,"r32":1,"freeheap":272172,"uptime":600} Mosquitto and another client sees same.

If my PubSubClient send this payload to TiniMQTT broker and another client (MQTT Explorer, MQTT Spy) sees other: {"r1":1,"r2":1,"r3":1,"r4":1,"r5":1,"r6":1,"r7":1,"r8":0,"r9":1,"r10":0,"r11":0,"r12":0,"r13":0,"r14":0,"r15":0,"r16":0,"r17":0,"r18":0,"r19":0,"r20":0,"r21":0,"r22":0,"r23":0,"r24":0,"r25":0,"r26":0,"r27":0,"r28":0,"r29":0,"r30":1,"r31":0,"r32":1,"freeheap":272172,"uptime":600}�

What is the ending symbol?

[pruwait]

remark: MQTT Explorer view mosquitto broker JSON payload, like multi-line, for example: { "r1": 1, "r2": 1, "r3": 1, "r4": 1, "r5": 1, "r6": 1, "r7": 1, "r8": 0, "r9": 1, "r10": 0, "r11": 0, "r12": 0, "r13": 0, "r14": 0, "r15": 0, "r16": 0, "r17": 0, "r18": 0, "r19": 0, "r20": 0, "r21": 0, "r22": 0, "r23": 0, "r24": 0, "r25": 0, "r26": 0, "r27": 0, "r28": 0, "r29": 0, "r30": 1, "r31": 0, "r32": 1, "freeheap": 270620, "uptime": 63 }

TiniMQTT broker send one-line payload with adding simbol. {"r1":1,"r2":1,"r3":1,"r4":1,"r5":1,"r6":1,"r7":1,"r8":0,"r9":1,"r10":0,"r11":0,"r12":0,"r13":0,"r14":0,"r15":0,"r16":0,"r17":0,"r18":0,"r19":0,"r20":0,"r21":0,"r22":0,"r23":0,"r24":0,"r25":0,"r26":0,"r27":0,"r28":0,"r29":0,"r30":1,"r31":0,"r32":1,"freeheap":270620,"uptime":63}�

[hsaturn]

Hello, thanks for the report. Obiously, the last character is a random char added. A reason could be that TinyMqtt pretends that the payload is one more byte than it is.

I'll check that very soon.

Thanks for the detailed report.

[hsaturn]

Last bug I've fixed was about long messages length decoding...

Funny thing, I've opened an issue : "Write unit test for length message decoding" ;-)

[hsaturn]

Hello

Just to tell that I keep in mind that issue. I'm very sorry for the long delay to fix this bug. I was very busy last weeks. I think I can take care of that issue this evening

Best regards

[hsaturn]

Hello

I've made severall tests. I've modified the library a little so that I can view Hexa dumps of what is sent / received.

I did not saw any problems with small payloads. (But I've found other bugs :-))

I'm going now to make some tests with a longer payloads.

[hsaturn]

And with a long payload, I can see now the spurious char

[hsaturn]

Hi Pruwait,

I thinkj I have a fix for this bug. I have to do some more tests because I have rewritten many part of the code (poor code on this part...).

[hsaturn]

Back at work on this problem.

[hsaturn]

My fix is working in one situation (yours). But unfortunately, I have some unit tests that are broken. So I can't release a new version yet.

[hsaturn]

Hello, back again I've released the version 0.8.0 of TinyMqtt that should fix your problem.

Feel free to reopen this issue if you still have some problem.

I wrote a unit test that is now passing successfully. I did not test the release on MqttBox I'll do that tomorrow when the release will be available directly in the Ardiuno IDE

Best Regards.

[hsaturn]

I'm reopening this topic. The bug is still there. The unit test is not going though any network interface. Message decoding is broken.

[hsaturn]

Hello, I'm verryyyy busy working on this issue. You may think this is a basic issue. And probably it is. But as I really want TinyMqtt to be a quality library, I do not want to quick fix this bug.

Instead, I want to write a unit test that fails, fix it and view the unit test green. This way, I'll be pretty sure that TinyMqtt will not suffer of regressions later.

Such a test has been written when clients do not send via wifi (but instead to another client of the same esp). But in order to write a unit test that sends through wifi, I need to emulate part of the ESPWifi library.

This is why this bug is long to be fixed. Other bugs will be faster but I have to finish the (EspMock for wifi.

Yet, I'm working on unit tests of .... esp mock, the lib that will allow better unit tests of TinyMqtt.

... Thousands of line of code for... probably two or three buggy lines :-D

Best regards

[hsaturn]

Hi all, some news.

I think I've finish the tcp/ip emulation. Basic tests with TinyMqtt are working with EspMock through virutal network.

I wrote a unit test for hudge payload (network_one_client_one_broker_hudge_publish_and_subscribe_through_network). But unexpectedly, this test passes !!!

That can mean three things 1 - EspMock (the virtual network) has any bug 2 - The test is not written correctly 3 - I was wrong when I though my bugfix was bad on TinyMqtt 4 - All is working as expected, I mean TinyMqtt is able to communicate to TinyMqtt with hudge payloads, but when we introduce some external Mqtt messages, things are starting to going wrong.

My intuitive guess is 4).

I'll check that tomorrow.

If you want to help me, You can clone the latest version from github (dfd5983715aaa...) and test it under your conditions.

Best regards

[terrorsl]

Hi, this code is not pretty, but it works for me. Hope it helps you.

int sMQTTMessage::encodeLength(char* msb, int length) const
{
	int count = 0;
	do
	{
		unsigned char encoded(length & 0x7f);
		length >>= 7;
		if (length) encoded |= 0x80;
		*msb++ = encoded;
		count++;
	} while (length);
	return count;
};
sMQTTError sMQTTMessage::sendTo(sMQTTClient *client,bool needRecalc)
{
	if (buffer.size())
	{
		if (needRecalc)
		{
			//debug("sending " << buffer.size() << " bytes");
			char *bytes = (char*)buffer.data();
			char size[4];
			size[0] = bytes[1];
			if (encodeLength(size, buffer.size() - vheader/*2*/) > 1)
			{
				buffer.insert(buffer.begin() + 2, size[1]);
			}
			buffer[1] = size[0];
		}
		// hexdump("snd");
		client->write(buffer.data(), buffer.size());
	}
	else
	{
		//debug("??? Invalid send");
		return sMQTTInvalidMessage;
	}
	return sMQTTOk;
};

[conaito]

@ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help

[terrorsl]

@ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help

What say compiler?

[conaito]

may you can provide your complete h and ccp file? i think this is much faster :) Am 19. Nov. 2021, 15:21 +0100 schrieb terrorsl @.***>:

@ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help What say compiler? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[terrorsl]

may you can provide your complete h and ccp file? i think this is much faster :) Am 19. Nov. 2021, 15:21 +0100 schrieb terrorsl @.***>: …

@ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help What say compiler? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

look here: https://github.com/terrorsl/sMQTTBroker sMQTTMessage.h and sMQTTMessage.cpp

[conaito]

very cool. thx! there no problems with longer messages? Am 21. Nov. 2021, 17:15 +0100 schrieb terrorsl @.***>:

may you can provide your complete h and ccp file? i think this is much faster :) Am 19. Nov. 2021, 15:21 +0100 schrieb terrorsl @.***>: …

@ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help What say compiler? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. look here: https://github.com/terrorsl/sMQTTBroker sMQTTMessage.h and sMQTTMessage.cpp — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[terrorsl]

very cool. thx! there no problems with longer messages? Am 21. Nov. 2021, 17:15 +0100 schrieb terrorsl @.***>: …

may you can provide your complete h and ccp file? i think this is much faster :) Am 19. Nov. 2021, 15:21 +0100 schrieb terrorsl @.***>: > … > > @ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help What say compiler? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. look here: https://github.com/terrorsl/sMQTTBroker sMQTTMessage.h and sMQTTMessage.cpp — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

It no problem for me, but need test.

[conaito]

i will for sure. is it based on this mqtt code or complete different ? Am 21. Nov. 2021, 18:15 +0100 schrieb terrorsl @.***>:

very cool. thx! there no problems with longer messages? Am 21. Nov. 2021, 17:15 +0100 schrieb terrorsl @.***>: …

may you can provide your complete h and ccp file? i think this is much faster :) Am 19. Nov. 2021, 15:21 +0100 schrieb terrorsl @.***>: > … > > @ terrorsl Help it to can get longer messages with your code? I try to add it into the cpp code but looks like need some more code adding (may also in header file?) Can you give me small idea to use it also? Thx for your help What say compiler? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. look here: https://github.com/terrorsl/sMQTTBroker sMQTTMessage.h and sMQTTMessage.cpp — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. It no problem for me, but need test. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[terrorsl]

some of the code are the same, but a lot is different.

[hsaturn]

Hello

The last git version (not yet the last release) should have fixed this. Best regards.

[hsaturn]

A unit test was added long ago (network_hudge_payload). TinyMqtt accept 32767 bytes payload (I guess) Test is on 210 bytes