MCU crash after few thousand TCP requests

[raphaelvalentin]

Hi,

I am encountering an issue where the MCU crashes after sending 80KB of data via TCP several thousand times. Below is a summary of the problem:

After several thousand successful data transfers, the TCP connection begins to fail with an "Out-of-order" packet (that can occur randomly let say between 1000 to 15000). This eventually leads to a complete crash of the MCU, rendering the device unresponsive and requiring a manual reset (if no watchdog).

Could you please help me investigate this behavior? It seems to be related to TCP buffer management or the handling of out-of-order packets within the Ethernet library or underlying stack. After trying to print some error logs, I feel out of options at this moment.

Board: Nucleo stm32f767zi All libraries: up-to-date

Thank you for your assistance!

• The arduino C/C++ reference code:

#include <LwIP.h>
#include <STM32Ethernet.h>

IPAddress ip(192, 168, 1, 196);
EthernetServer server(80);

void sendChunkedData(EthernetClient& client,
                     const unsigned char* data,
                     unsigned int length) {
    const unsigned int chunkSize = 256;
    for (unsigned int i = 0; i < length; i += chunkSize) {
        unsigned int remaining = length - i;
        unsigned int sizeToSend =
            (remaining < chunkSize) ? remaining : chunkSize;
        size_t sent = client.write(data + i, sizeToSend);
        if (sent != sizeToSend) {
            Serial.println("error size");
            Serial.flush();
            return;
        }
    }
}

// Generates Lorem Ipsum text into the provided buffer and returns the length of the generated string
size_t generateLoremIpsum(char *buffer, size_t sizeInBytes) {
    const char *lorem = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. "
                        "Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. "
                        "Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. "
                        "Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. "
                        "Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ";
    
    size_t loremLen = strlen(lorem);
    size_t pos = 0;

    while (pos + loremLen < sizeInBytes - 1) {
        strcpy(buffer + pos, lorem);
        pos += loremLen;
    }
    
    // Copy the remaining part to fill up the buffer if there's any space left
    if (pos < sizeInBytes - 1) {
        strncpy(buffer + pos, lorem, sizeInBytes - 1 - pos);
        pos += (sizeInBytes - 1 - pos); // Update position after the last part is copied
    }

    buffer[pos] = '\0'; // Ensure null termination
    return pos;         // Return the length of the generated string
}

const size_t myText_size = 80000;
size_t myText_length = 0;
uint8_t myText[myText_size];

void setup() {
    // Open serial communications and wait for port to open:
    Serial.begin(9600);
    while (!Serial) {
        ;  // wait for serial port to connect. Needed for native USB port only
    }
    myText_length  = generateLoremIpsum((char *) myText, myText_size);
    Ethernet.begin(ip);
    server.begin();
    Serial.print("server is at ");
    Serial.println(Ethernet.localIP());
    Serial.print("length of text: ");
    Serial.println(myText_length);
    Serial.print("sizeof of text: ");
    Serial.println(sizeof(myText));
}

void loop() {
    EthernetClient client = server.available();
    if (client) {

        bool currentLineIsBlank = true;
        // delay(5);
        while (client.connected()) {
            //delay(1);
            if (client.available()) {
                //delay(1);
                char c = client.read();

                if (c == '\n' && currentLineIsBlank) {
                    // delay(1);
                    client.println("HTTP/1.1 200 OK");
                    client.println("Content-Type: text/plain");
                    client.print("Content-Length: ");
                    client.println(myText_length);
                    client.println("Connection: close");
                    client.println();
                    sendChunkedData(client, (const unsigned char *) myText, myText_length);
                    // client.write((const char *) myText, myText_length);
                    break;
                }
                if (c == '\n') {
                    currentLineIsBlank = true;
                } else if (c != '\r') {
                    currentLineIsBlank = false;
                }
            }
        }
        delay(5);
        client.stop();
        delay(1);
    }
}

• The bash script:

#!/bin/bash

# URL of the file to download
URL="http://192.168.1.196/"

# Number of times to download the file
ITERATIONS=10000

# Loop to download the file multiple times
for ((i=1; i<=ITERATIONS; i++))
do
  echo "Downloading file $i of $ITERATIONS..."
  wget -T 30 -O "mytext.txt" $URL
  sleep 0.01

  # Check if the download was successful
  if [[ $? -ne 0 ]]; then
    echo "Download failed at iteration $i"
    break
  fi
done

echo "Download completed."

• The log file from the bash script:

2024-08-17 12:20:29 (7.42 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4163 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79999 (78K) [text/plain]
Saving to: ‘mytext.txt’

mytext.txt                          100%[=================================================================>]  78.12K  --.-KB/s    in 0.01s   

2024-08-17 12:20:29 (6.80 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4164 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79999 (78K) [text/plain]
Saving to: ‘mytext.txt’

mytext.txt                          100%[=================================================================>]  78.12K  --.-KB/s    in 0.01s   

2024-08-17 12:20:29 (6.80 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4165 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... failed: Connection timed out.
Retrying.

--2024-08-17 12:21:00--  (try: 2)  http://192.168.1.196/
Connecting to 192.168.1.196:80... ^C

real	4m16.630s
user	0m31.303s
sys	1m33.087s

• The Wireshark screenshot:

A Wireshark capture shows that a TCP "Out-of-order" event occurs right before the MCU crashes. This event might indicate a problem in the TCP stack or network buffer handling (lwip parameters not optimized, memory corruption or race condition?).