Strange behavior on long payloads

[Jobenas]

Hi,

I'm using your library to publish payloads that are of about 200 characters long. I've modified the MQTT_MAX_PACKET_SIZE on the PubSubClient.h file and I'm able to publish these messages as I'm seeing them on a sub that I set up on Python. The problem I'm getting is that some of the messages are not arriving properly. For instance the type of message I should be seeing is {"operation":"set","type":"data","sender":"jobenas/esptest","value":"{"owner":"jobenas/esptest","entryType":"data","value":766,"units":"none","datatype":"integer"}"}, which I get sometimes but some other times I get complete garbage like: {"op� @0, I haven't detected a pattern yet as to when I get garbage and when I get the correct message.

I'm using the ESP8266 directly and using the arduino core v 2.3 and the PubSubClient library v 2.6. I will be looking at the packets as I get them to see if there's any clue there but any solution or pointers to the problem would be deeply welcomed. Thanks in advance.

Jorge.

PS: I'm pasting my code below.

`#include <ESP8266WiFi.h> #include <PubSubClient.h>

#include <ArduinoJson.h>

// Update these with values suitable for your network.

const char* ssid = "BANET"; const char* password = "4271051500"; const char* mqtt_server = "ec2-52-27-179-130.us-west-2.compute.amazonaws.com";

WiFiClient espClient; PubSubClient client(espClient); long lastMsg = 0; //char msg[250]; int value = 0;

void setup() { pinMode(BUILTIN_LED, OUTPUT); // Initialize the BUILTIN_LED pin as an output Serial.begin(115200); setup_wifi(); client.setServer(mqtt_server, 1883); client.setCallback(callback); }

void setup_wifi() {

delay(10); // We start by connecting to a WiFi network Serial.println(); Serial.print("Connecting to "); Serial.println(ssid);

WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); }

Serial.println(""); Serial.println("WiFi connected"); Serial.println("IP address: "); Serial.println(WiFi.localIP()); }

void callback(char* topic, byte* payload, unsigned int length) { Serial.print("Message arrived ["); Serial.print(topic); Serial.print("] "); for (int i = 0; i < length; i++) { Serial.print((char)payload[i]); } Serial.println();

// Switch on the LED if an 1 was received as first character if ((char)payload[0] == '1') { digitalWrite(BUILTIN_LED, LOW); // Turn the LED on (Note that LOW is the voltage level // but actually the LED is on; this is because // it is acive low on the ESP-01) } else { digitalWrite(BUILTIN_LED, HIGH); // Turn the LED off by making the voltage HIGH }

}

void reconnect() { // Loop until we're reconnected while (!client.connected()) { Serial.print("Attempting MQTT connection..."); // Attempt to connect if (client.connect("ESP8266Client")) { Serial.println("connected"); // Once connected, publish an announcement... client.publish("outTopic", "hello world"); // ... and resubscribe client.subscribe("inTopic"); } else { Serial.print("failed, rc="); Serial.print(client.state()); Serial.println(" try again in 5 seconds"); // Wait 5 seconds before retrying delay(5000); } } } void loop() {

if (!client.connected()) { reconnect(); } client.loop();

long now = millis(); if (now - lastMsg > 5000) { lastMsg = now; ++value; char* body = generateDataBody(value); char* format = generateDataFormat(body); // snprintf (msg, 75, "hello world #%ld", value); Serial.print("Publish message: "); Serial.println(format); client.publish("topic/test", format); } }

char* generateDataBody(int value) { StaticJsonBuffer<200> jsonBuffer;

JsonObject& message = jsonBuffer.createObject(); message["owner"] = "jobenas/esptest"; message["entryType"] = "data"; message["value"] = value; message["units"] = "none"; message["datatype"] = "integer";

char bodyString[200]; message.printTo(bodyString, sizeof(bodyString));

return bodyString; }

char* generateDataFormat(char* body) { StaticJsonBuffer<300> jsonBuffer;

JsonObject& message = jsonBuffer.createObject(); message["operation"] = "set"; message["type"] = "data"; message["sender"] = "jobenas/esptest"; message["value"] = body;

char dataString[300]; message.printTo(dataString, sizeof(dataString));

return dataString; }`

[Jobenas]

Just in case anyone else is having the same problem, I managed to solved it by doing two things:

First I included a new function in the library that accepts the topic, payload and the length. I discovered that for some reason when providing three arguments to the publish function it always defaulted to the third argument being retained (I don't know if the reason and I'm not that fluent in C++), that didn't solve the problem since I still received some payloads with garbage and a smaller length.

The second thing I did was I took out the strlen(payload) from the function call and assigned them to a variable that was then passed to the arguments of the function, that way I got the correct payload 100% of the time. I'm guessing this issue could be due to a stack overflow or something like that.

Anyway that allowed me to send payloads of about 200 bytes although I changed the definition to 500, which in theory should work. Hope this helps anyone that's running into the same problems.

[luisgcu]

hello I have similar problem publishing long payloads. StaticJsonBuffer<200> jsonBuffer; JsonObject& root = jsonBuffer.createObject(); root["Lat"] = mobile4data.latitude; root["Lon"] = mobile4data.latitude; root["Volt"] = mobile4data.voltage; root["CTemp"] = mobile4data.coolantTemp; root["FPres"]= mobile4data.fuelPressure; root["EngRpm"] = mobile4data.engineRPM; root["VSpeed"] = mobile4data.vehicleSpeed; root["FLev"] = mobile4data.fuelLevel; root["test"] = 12345; root["AbTemp"] = mobile4data.ambientTemp; root.printTo(Serial); Serial.println(); root.printTo(buffer, sizeof(buffer)); //root.prettyPrintTo(Serial); client.publish("outTopic", buffer); I did the same as you increasing the MQTT_MAX_PACKET_SIZE on the PubSubClient.h to 256 and no luck. if i remove one data element from the JasonObject then is published ok. Any ideas of what I can do?

[Jobenas]

I believe the problem lies in the fact that when you use the publish method with just two parameters, the length of the payload is calculated at that point (seeing the library when you use publish with two parameters it passes those two parameters plus strlen(payload) to the function that actually does the work), and something with the stack (don't want to say overflow because the code still works after the function finishes). That is why it probably works with small values and bigger ones just generate garbage.

My solution, as I pointed above was to create a new method so I could directly pass the length (for some reason, probably my inexperience with C++), the thing is that you also need to calculate the length outside the method before you call it. I declared something like this:

int length = strlen(payload); client.publish(topic, payload, length);

Hope it works for you.

[luisgcu]

thanks so much for your time and advice, but that didn't work either, can see the payload when printing to serial . but it wont go out with the client.publish If I remove just one element from the json object then the client.publish work fine. {"Lat":80.12,"Lon":80.12,"Volt":12.33,"CTemp":91,"FPres":123,"EngRpm":912,"VSpeed":67,"FLev":20,"test":12345,"AbTemp":33}

[Jobenas]

have you checked if your payload length is lower than 256 bytes? that could be the source of your problem.

[gilles906]

Hi What can I do if my payload length is over than 256 bytes, like 320bytes? Thanks for answer Gilles

[marciopamplona]

I have the same problem, my message is cutted about 541~530 bytes. I tried to use the publish(status.c_str(), (uint8_t*)temp.c_str(), temp.length(), false) , same result.

[Nitinrajyadav]

Did anyone find some workaround? Need help....

[Nitinrajyadav]

on line 26 PubSubClient.h
#define MQTT_MAX_PACKET_SIZE 256

on line 132 PubSubClient.h
boolean publish(const char* topic, const char* payload, unsigned int length); // new

on line 355 PubSubClient.cpp

boolean PubSubClient::publish(const char* topic, const char* payload, unsigned int length) { return publish(topic,(const uint8_t*)payload, length, false); }

and then

jsonReply.printTo(msg_buff_200, sizeof(msg_buff_200)); unsigned int length = strlen(msg_buff_200); mqttClient.publish(mqtt_outgoing_topic, msg_buff_200, length);

thanks @Jobenas

[eueduardo3]

Muito obrigado @Nitinrajyadav

[wbuh123]

@Nitinrajyadav thank you! Works perfect! Can't understand why this (from 2018) still not found it's way into the official release (2.7)? Isn't it maintained anymore?

[knolleary]

I'm not sure the explanation holds up.

If I generate a large payload with something like this:

int size = 1000;
char msg[1000];

for (int i=0;i<size;i++) {
   msg[i] = 'a'+(i%26);
}
msg[size-1] ='!';
msg[size] = 0;

client.publish("outTopic", msg);

I can publish is without any corruption. I have scoured the code for issues around the strlen() handling of char* payloads - I cannot find anything in there that would explain the issue.

When I try the code from the original issue, that uses StaticJsonBuffer and others to generate the payload, I do routinely see corrupted data. However, when I debug the payload before it gets passed to the publish function, I see it is corrupt before the call to publish is even made.

That points to some issue with the use of StaticJsonBuffer - that isn't a library I'm familiar with and I see it has been substantially rewritten in its 6.x version (you need to go back to 5.x to compile the code shared above).

@wbuh123 given you appear to be hitting this more recently then the original report, can you share any details on how you are generating your payload?

[arthurnni]

I also have the same problem. in my case how to fix this problem is to publish after this reconnect code .

if (!client.connected()) { reconnect(); } client.publish("outTopic", long_payload);