ESP32-S3 - Arduino v3.1 - USBCDC Hangs and misses chunks of data for for large serial transmits

[TheCrypt0]

Board

ESP32-S3

Device Description

Custom ESP32-S3 keyboard but it should work on any ESP32-S3 devkit with USB connection.

Hardware Configuration

N.A.

Version

v3.1.0

IDE Name

Arduino IDE

Operating System

macOS

Flash frequency

80MHz

PSRAM enabled

no

Upload speed

921600

Description

I’m currently porting my project to the latest version of the Arduino Core but I'm experiencing data loss when using the USBCDC implementation with large chunks of data. It seems that the data is not reliably received and a buffer overflow occurs.

This seems to be related and exactly what I am experiencing: https://www.esp32.com/viewtopic.php?t=38858

Downgrading to 2.0.17 makes the code work again.

I suspect the issue might be related to how data is queued or buffered internally within the USBCDC implementation. If there are known limitations or configurations that could mitigate this issue, I’d really appreciate any guidance or suggestion.

I've provided an example that computes the SHA1 of the sent data to confirm integrity.

Sketch

#include <Arduino.h>
#include <ArduinoBearSSL.h>

#include <USB.h>
#include <USBCDC.h>

USBCDC         m_serial;
ESPUSB*        m_usb;

String m_buffer;

int m_last_message_ms;

void ingest_cdc_data(void)
{
  m_serial.println("GOT DATA");

  while (m_serial.available()) {
    m_buffer += m_serial.readString();
  }

  if (m_buffer.length() == 0) {
    return;
  }

  m_last_message_ms = millis();
}

void calculate_buffer_sha1()
{
  if(m_buffer == "")
    return;

  // calculate the sha1 of the buffer if it's been more than 1 second since the last message
  if (millis() - m_last_message_ms > 1000) 
  {
    SHA1.beginHash();
    SHA1.print(m_buffer);
    SHA1.endHash();

    print_hash();

    m_buffer = "";
  }
}

void print_hash()
{
  while (SHA1.available()) {
    byte b = SHA1.read();

    if (b < 16) {
      m_serial.print("0");
    }

    m_serial.print(b, HEX);
  }
  m_serial.println();
}

void
usb_event_callback(void*            arg,
                 esp_event_base_t event_base,
                 int32_t          event_id,
                 void*            event_data)
{
  if (event_base == ARDUINO_USB_CDC_EVENTS) {
    switch (event_id) {
      case ARDUINO_USB_CDC_RX_EVENT:
        ingest_cdc_data();
        break;
      case ARDUINO_USB_CDC_RX_OVERFLOW_EVENT:
        m_serial.printf("USB cdc overflow\n");
        break;

      default:
        break;
    }
  }
}

void
setup()
{
  // put your setup code here, to run once:
  m_buffer = "";
  m_last_message_ms = 0;

  USB.onEvent(usb_event_callback);
  m_serial.onEvent(usb_event_callback);

  m_serial.begin(115200);

  m_usb = &USB; // get the USB object

  m_usb->begin();
}

void
loop()
{
  // put your main code here, to run repeatedly:
  calculate_buffer_sha1();

  delay(1);
}

Debug Message

--- CORRECT OUTPUT WITH ARDUINO ESP32 VER 2.0.17 ---
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
GOT DATA
128DA61AA85A72155E5229129D83E6044181FB0A

--- OUTPUT WITH ARDUINO ESP32 VER 3.1.0 ---
GOT DATA
GOT DATA
GOT DATA
(then it stops and doesn't accept any more data)

Other Steps to Reproduce

Open an Arduino Serial monitor and paste the following text into the Message box (No Line Ending, 115200 baud).

111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

Expect the SHA1 being: 128DA61AA85A72155E5229129D83E6044181FB0A

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

@SuGlider can look into this, but could you try with the HWCDC instead? If Serial is all you need, HWCDC might do better

[TheCrypt0]

@me-no-dev thanks for the reply!

Is there a way to use USBHIDKeyboard and the other HID peripherals along with HWCDC?

[SuGlider]

@me-no-dev thanks for the reply!

Is there a way to use USBHIDKeyboard and the other HID peripherals along with HWCDC?

Nop. HWCDC is exclusivelly a CDC/JTAG implementation in silicon. It can't work in any other way.

ESP32-S3 has USB OTG mode for a full USB Class mode software implementation. Arduino uses TinyUSB implementation within its USB Class.

[TheCrypt0]

Nop. HWCDC is exclusivelly a CDC/JTAG implementation in silicon. It can't work in any other way.

ESP32-S3 has USB OTG mode for a full USB Class mode software implementation. Arduino uses TinyUSB implementation within its USB Class.

@SuGlider

Yeah, that's what I was worried about, I have to use the USB OTG mode. At least we have full control via software on what happens.

Have you managed to replicate the issue with the instructions provided or is there something I can do meanwhile to help you guys with the process? I'll be happy to provide support if able.

[SuGlider]

@TheCrypt0 - Thanks for the support. I'll set up an environment for replicate and debug the issue.

[EarlVadim]

In my project I use HWCDC, and sometimes it also misses packets. ((( There is some problem there.

[TheCrypt0]

Hey @SuGlider apologies for bothering you again, have you been able to replicate the issue? It seems like it's affecting others as well.

Thanks!

[SuGlider]

Hey @SuGlider apologies for bothering you again, have you been able to replicate the issue? It seems like it's affecting others as well.

Thanks!

Hi, I'm on vacation right now and I shall return to work in the beginning of February. This issue is in top place for investigation.

[TheCrypt0]

Hey, @SuGlider have you got any news? Is there any way I can help with this? Thank you!

[raccoon-forvia]

have same issue as you do, that it will hangs with no reason and can't recover

[SuGlider]

Problem has been found in Tiny USB DCD layer. Currently working on it.

[TD-er]

Ohh great! Looking forward testing it.