nuttx icon indicating copy to clipboard operation
nuttx copied to clipboard

[HELP] ESP32-S3 UART repeats stale data after long uptime

Open thiagofinelon opened this issue 2 months ago • 8 comments

Description

Summary I’m running a long-lived application on an ESP32-S3 using the current NuttX master. After an unpredictable amount of time—sometimes minutes, sometimes hours—the UART stops behaving correctly: incoming traffic seems to replay older bytes (e.g., the tail of a previous AT response) instead of delivering the newest characters. The external device is still transmitting, so it looks as if the driver is serving a stale buffer.

Environment

SoC: ESP32-S3 (esp32s3-devkit) UART: 115200 bps, 8N1. Workload: internal modem/PPP service that keeps the link busy (logs + AT commands) Observed Behavior

System boots and UART traffic flows normally. After a random interval, reads begin returning bytes that were already consumed earlier (“ghost” data). From that moment on, the stream never delivers new characters, effectively stalling the higher-level protocol. Only a full system reset recovers the port. Hypothesis: the RX FIFO overflows, leaving the hardware/driver in a wedged state with old bytes queued. Mitigation Attempt To explore the issue, I tweaked arch/xtensa/src/esp32s3/esp32s3_serial.c:

In the ISR I now watch UART_RXFIFO_OVF_INT_ST_M; when set, I call esp32s3_lowputc_rst_rxfifo(priv) before clearing the status, hoping to drop corrupted data. Raised the RX “FIFO full” threshold to 8 bytes so bursts don’t trigger an interrupt per byte. Enabled RX timeout with a threshold of 10, so shorter frames still wake the ISR even if they never fill the FIFO. These changes seem to postpone the failure but I’m not yet convinced they fully solve it.

Questions for the maintainers

Does resetting the FIFO on overflow and adjusting the thresholds sound like the right approach, or am I just hiding the real root cause? Are there recommended settings or patches for handling long-term, high-volume UART traffic on the ESP32-S3 under NuttX?

Alterations in esp32s3_serial.c : uart_handler()

/* ISR additions */

uint32_t rx_ovf_mask = UART_RXFIFO_OVF_INT_ST_M;
...
if ((int_status & rx_mask) != 0)
  {
    uart_recvchars(dev);
    modifyreg32(UART_INT_CLR_REG(priv->id), rx_mask, rx_mask);
  }

if ((int_status & rx_ovf_mask) != 0)
  {
    esp32s3_lowputc_rst_rxfifo(priv);
    modifyreg32(UART_INT_CLR_REG(priv->id), rx_ovf_mask, rx_ovf_mask);
  }

Alterations in esp32s3_serial.c : esp32s3_setup()

static int esp32s3_setup(struct uart_dev_s *dev)
{
  struct esp32s3_uart_s *priv = dev->priv;
  const uint32_t ticks_tout = 10U;
  const uint32_t rx_fifo_thrhd = 8U;
  ...
  modifyreg32(UART_CONF1_REG(priv->id), UART_RXFIFO_FULL_THRHD_M,
              rx_fifo_thrhd << UART_RXFIFO_FULL_THRHD_S);

  modifyreg32(UART_CONF1_REG(priv->id),
              UART_RX_TOUT_THRHD_M | UART_RX_TOUT_EN_M,
              (ticks_tout << UART_RX_TOUT_THRHD_S) |
              UART_RX_TOUT_EN_M);
  ...
}

Verification

  • [x] I have verified before submitting the report.

thiagofinelon avatar Oct 09 '25 13:10 thiagofinelon

Hi, Thanks for the bug report. Could give some information about your setup? For example

  • NuttX version that running on device
  • Defconfig that is being used
  • Doesn't have to code itself but some code snippet that we can recreate issue on our side would be nice.

Clearing up fifo buffer might cause data loss too. For now increasing RX buffer size with CONFIG_UARTX_RXBUFSIZE could help the issue or at least worth to try.

eren-terzioglu avatar Oct 09 '25 13:10 eren-terzioglu

@eren-terzioglu

Im using 12.09 nuttx. I cant disclouse my application, but Im using pppd + chat, like the example.

CONFIG_ALLOW_BSD_COMPONENTS=y CONFIG_ARCH="xtensa" CONFIG_ARCH_BOARD="esp32s3-devkit" CONFIG_ARCH_BOARD_COMMON=y CONFIG_ARCH_BOARD_ESP32S3_DEVKIT=y CONFIG_ARCH_CHIP="esp32s3" CONFIG_ARCH_CHIP_ESP32S3=y CONFIG_ARCH_CHIP_ESP32S3WROOM2N16R8V=y CONFIG_ARCH_INTERRUPTSTACK=2048 CONFIG_ARCH_STACKDUMP=y CONFIG_ARCH_XTENSA=y CONFIG_AT_INTERFACE=y CONFIG_BOARDCTL_RESET=y CONFIG_BOARD_LATE_INITIALIZE=y CONFIG_BOARD_LOOPSPERMSEC=16717 CONFIG_BUILTIN=y CONFIG_CRYPTO=y CONFIG_CRYPTO_MBEDTLS=y CONFIG_CRYPTO_RANDOM_POOL=y CONFIG_DEBUG_FULLOPT=y CONFIG_DEBUG_SYMBOLS=y CONFIG_DEFAULT_TASK_STACKSIZE=4096 CONFIG_DEV_GPIO=y CONFIG_DEV_URANDOM=y CONFIG_DRIVERS_IEEE80211=y CONFIG_DRIVERS_WIRELESS=y CONFIG_ESP32S3_EFUSE=y CONFIG_ESP32S3_FLASH_MODE_QIO=y CONFIG_ESP32S3_MERGE_BINS=y CONFIG_ESP32S3_RT_TIMER_TASK_STACK_SIZE=4096 CONFIG_ESP32S3_SPI2=y CONFIG_ESP32S3_SPIFLASH=y CONFIG_ESP32S3_SPIFLASH_LITTLEFS=y CONFIG_ESP32S3_UART0=y CONFIG_ESP32S3_UART0_RXPIN=43 CONFIG_ESP32S3_UART0_TXPIN=44 CONFIG_ESP32S3_UART1=y CONFIG_ESP32S3_UART1_RXPIN=9 CONFIG_ESP32S3_UART1_TXPIN=5 CONFIG_ESPRESSIF_POWER_SAVE_MAX_MODEM=y CONFIG_ESPRESSIF_WIFI=y CONFIG_ESPRESSIF_WIFI_STATION_SOFTAP=y CONFIG_ESP_WPA_DEBUG_PRINT=y CONFIG_EXAMPLES_DHCPD=y CONFIG_EXAMPLES_RANDOM=y CONFIG_FS_PROCFS=y CONFIG_FS_SPIFFS=y CONFIG_FS_TMPFS=y CONFIG_IDLETHREAD_STACKSIZE=8046 CONFIG_INIT_ENTRYNAME="" CONFIG_INIT_ENTRYPOINT="" CONFIG_INIT_STACKSIZE=8192 CONFIG_INTELHEX_BINARY=y CONFIG_IOB_NBUFFERS=124 CONFIG_IOB_THROTTLE=24 CONFIG_LIBC_EXECFUNCS=y CONFIG_LIBC_MEMFD_ERROR=y CONFIG_LIBC_PERROR_STDOUT=y CONFIG_LIBC_STRERROR=y CONFIG_LIBC_STRERROR_ERRNUM=y CONFIG_LINE_MAX=64 CONFIG_MBEDTLS_ENTROPY_HARDWARE_ALT=y CONFIG_MCA_GPIO_EN_HOTSPOT_WIFI="/dev/gpio7" CONFIG_MCA_MAIN_APP=y CONFIG_MCA_MAIN_APP_DEBUG=y CONFIG_MCA_MAIN_APP_STACKSIZE=16384 CONFIG_MODEM_BACKEND=y CONFIG_MODEM_HANDLER=y CONFIG_MQ_MAXMSGSIZE=512 CONFIG_NAME_MAX=48 CONFIG_NETDB_DNSCLIENT=y CONFIG_NETDB_DNSSERVER_IPv4ADDR=0x08080808 CONFIG_NETDEV_LATEINIT=y CONFIG_NETDEV_PHY_IOCTL=y CONFIG_NETDEV_WIRELESS_IOCTL=y CONFIG_NETDEV_WORK_THREAD=y CONFIG_NETINIT_DHCPC=y CONFIG_NETINIT_WAPI_PASSPHRASE="" CONFIG_NETINIT_WAPI_SSID="" CONFIG_NETUTILS_DHCPD=y CONFIG_NETUTILS_DHCPD_SIGWAKEUP=10 CONFIG_NETUTILS_IPERF=y CONFIG_NETUTILS_WEBCLIENT=y CONFIG_NET_BROADCAST=y CONFIG_NET_ETH_PKTSIZE=1514 CONFIG_NET_ICMP_SOCKET=y CONFIG_NET_TCP=y CONFIG_NET_TCPBACKLOG=y CONFIG_NET_TCP_DELAYED_ACK=y CONFIG_NET_TCP_KEEPALIVE=y CONFIG_NET_TCP_WRITE_BUFFERS=y CONFIG_NET_UDP=y CONFIG_NET_UDP_WRITE_BUFFERS=y CONFIG_NFILE_DESCRIPTORS_PER_BLOCK=12 CONFIG_NSH_ARCHINIT=y CONFIG_NSH_BUILTIN_APPS=y CONFIG_NSH_FILEIOSIZE=512 CONFIG_NSH_MOTD=y CONFIG_NSH_MOTD_STRING="" CONFIG_NSH_PROMPT_MAX=256 CONFIG_NSH_PROMPT_STRING="" CONFIG_NSH_READLINE=y CONFIG_NSH_STRERROR=y CONFIG_PIPES=y CONFIG_PM=y CONFIG_PM_ALARM_SEC=1200 CONFIG_PM_EXT1_WAKEUP=y CONFIG_PM_EXT1_WAKEUP_RTC_GPIO7=y CONFIG_PM_GOVERNOR_EXPLICIT_RELAX=-1 CONFIG_PM_GOVERNOR_GREEDY=y CONFIG_PM_NDOMAINS=3 CONFIG_PM_SLEEP_WAKEUP_SEC=60 CONFIG_POSIX_SPAWN_DEFAULT_STACKSIZE=2048 CONFIG_PREALLOC_TIMERS=4 CONFIG_PTHREAD_MUTEX_TYPES=y CONFIG_RAM_SIZE=114688 CONFIG_RAM_START=0x20000000 CONFIG_READLINE_CMD_HISTORY=y CONFIG_READLINE_TABCOMPLETION=y CONFIG_RR_INTERVAL=10 CONFIG_SCHED_BACKTRACE=y CONFIG_SCHED_LPWORK=y CONFIG_SERVICE_LOGGER=y CONFIG_SERVICE_LOGGER_LEVEL_N0=y CONFIG_SERVICE_MQTT=y CONFIG_SERVICE_MQTT_AUTH=y CONFIG_SERVICE_MQTT_AUTH_PASS="" CONFIG_SERVICE_MQTT_AUTH_USER="" CONFIG_SERVICE_MQTT_HOST="" CONFIG_SERVICE_MQTT_HOST_PORT="" CONFIG_SERVICE_MQTT_RXSIZE=512 CONFIG_SERVICE_MQTT_SYNC_TIME=200 CONFIG_SERVICE_MQTT_TXSIZE=1024 CONFIG_SERVICE_SAT=y CONFIG_SERVICE_SAT_STX3_CTS="/dev/gpio5" CONFIG_SERVICE_SAT_STX3_DEBUG=y CONFIG_SERVICE_SAT_STX3_RST="/dev/gpio0" CONFIG_SERVICE_SAT_STX3_RTS="/dev/gpio1" CONFIG_SERVICE_SAT_STX3_UART="/dev/ttyS0" CONFIG_SERVICE_WIFI=y CONFIG_SERVICE_WIFI_AP=y CONFIG_SERVICE_WIFI_AP_GATEWAY="" CONFIG_SERVICE_WIFI_AP_IP="" CONFIG_SERVICE_WIFI_AP_PASS="" CONFIG_SERVICE_WIFI_AP_SSID="" CONFIG_SERVICE_WIFI_DEBUG=y CONFIG_SERVICE_WIFI_STA=y CONFIG_SIG_DEFAULT=y CONFIG_SIM7080G_BACKEND_DTR="/dev/gpio4" CONFIG_SIM7080G_BACKEND_PWR="/dev/gpio3" CONFIG_SIM7080G_BACKEND_UART="/dev/ttyS1" CONFIG_SIM7080G_GNSS_TIMEOUT_MINUTES=5 CONFIG_SIM7080G_REGISTRATION_TIMEOUT_MINUTES=5 CONFIG_SMP=y CONFIG_SMP_NCPUS=2 CONFIG_SPIFFS_NAME_MAX=128 CONFIG_SPI_DRIVER=y CONFIG_SPI_SLAVE=y CONFIG_SPI_SLAVE_DRIVER=y CONFIG_STACK_COLORATION=y CONFIG_STACK_USAGE=y CONFIG_START_DAY=6 CONFIG_START_MONTH=12 CONFIG_START_YEAR=2011 CONFIG_SYSLOG_BUFFER=y CONFIG_SYSTEM_CLE=y CONFIG_SYSTEM_DHCPC_RENEW=y CONFIG_SYSTEM_NSH=y CONFIG_SYSTEM_PING=y CONFIG_SYSTEM_POPEN=y CONFIG_THREAD_MQTT_QUEUE_STACK_SIZE=8192 CONFIG_THREAD_MQTT_STACK_SIZE=8192 CONFIG_THREAD_SPI_STACK_SIZE=8192 CONFIG_TIMER=y CONFIG_TLS_TASK_NELEM=4 CONFIG_UART0_BAUD=9600 CONFIG_UART1_RXBUFSIZE=512 CONFIG_UART1_TXBUFSIZE=512 CONFIG_WEBCLIENT_MAXFILENAME=64 CONFIG_WEBCLIENT_MAXHTTPLINE=1024 CONFIG_WEBCLIENT_MAXMIMESIZE=64 CONFIG_WEBCLIENT_TIMEOUT=30 CONFIG_WIRELESS=y CONFIG_WIRELESS_WAPI=y CONFIG_WIRELESS_WAPI_CMDTOOL=y CONFIG_WIRELESS_WAPI_INITCONF=y CONFIG_WIRELESS_WAPI_STACKSIZE=8192

thiagofinelon avatar Oct 09 '25 13:10 thiagofinelon

Hi, Thanks for the bug report. Could give some information about your setup? For example

  • NuttX version that running on device
  • Defconfig that is being used
  • Doesn't have to code itself but some code snippet that we can recreate issue on our side would be nice.

Clearing up fifo buffer might cause data loss too. For now increasing RX buffer size with CONFIG_UARTX_RXBUFSIZE could help the issue or at least worth to try.

I already tried to tweak CONFIG_UARTX_RXBUFSIZE, didnt worked.

thiagofinelon avatar Oct 09 '25 13:10 thiagofinelon

Hi, Thanks for the bug report. Could give some information about your setup? For example

  • NuttX version that running on device
  • Defconfig that is being used
  • Doesn't have to code itself but some code snippet that we can recreate issue on our side would be nice.

Clearing up fifo buffer might cause data loss too. For now increasing RX buffer size with CONFIG_UARTX_RXBUFSIZE could help the issue or at least worth to try.

I just reset rx fifo, when a rx overflow occurs, this is the same implementation of esp idf uart.

thiagofinelon avatar Oct 09 '25 13:10 thiagofinelon

Seems it is pretty close to IDF implementation, please create a PR and we can test feature more.

eren-terzioglu avatar Oct 10 '25 07:10 eren-terzioglu

Perhaps CDC-ACM USB serial have the same issue. It seems to be stuck after receiving lot of data.

w2016561536 avatar Oct 16 '25 14:10 w2016561536

@w2016561536 Can you share a minimal setup do reproduce this? Maybe I can debug this also

thiagofinelon avatar Oct 16 '25 14:10 thiagofinelon

@w2016561536 Can you share a minimal setup do reproduce this? Maybe I can debug this also

Hi, I found this bug while running PX4-Autopilot. Use usb serial to transmit mavlink message in a rate of 3KB/s. Mcu tx will fail in several minutes. I am trying to reproduce it without PX4. BTW, your patch works well in serial fix. Thanks! So i guess the usb serial may have similar problem.

w2016561536 avatar Oct 16 '25 16:10 w2016561536