nuttx icon indicating copy to clipboard operation
nuttx copied to clipboard
verified publisher icon apache

[BUG] RP2040 ostest fail when usb data is connected

Open keever50 opened this issue 8 months ago • 9 comments

Description / Steps to reproduce the issue

Any RP2040 based boards are not passing the OS test for me. The OS test hangs at the watchdog test. This happens when the RP2040 based boards are connected to USB. They do not have to be powered by the USB. It is specifically when they are connected to it.

Crashes depend on what machine its connected to:

  • Crashes on my desktop. Windows and Linux.
  • Crashes on my macbook. MacOS
  • --Runs fine on my Framework laptop 13. Windows and Linux. (How strange)-- (Not anymore)

Crashes do not depend on:

  • Power supply
  • NuttX configuration (so far i can test)
  • Board configuration, both config and physical
  • USB cable

To reproduce a normal run without crash:

  1. Configure a NSH configuration over UART.
  2. Connect a UART reader of choice
  3. DO NOT connect USB.
  4. Connect an external power supply
  5. Run ostest
  6. It should run fine

To reproduce a crash:

  1. Configure either NSH or USBNSH config
  2. Connect USB (including the datalines)
  3. Read out the RP2040 with either USB or UART reader.
  4. Run ostest
  5. Should crash around the watchdog test

Cross testing: I could not replicate this on any other microcontroller I have. Most testing was done on the STM32F4 and STM32H7.

On which OS does this issue occur?

[OS: Other]

What is the version of your OS?

Latest

NuttX Version

Master

Issue Architecture

[Arch: arm]

Issue Area

[Area: Kernel]

Host information

N/A

Verification

  • [x] I have verified before submitting the report.

keever50 avatar Apr 05 '25 11:04 keever50

Additional info about RP2040 boards: I have tried the Raspberry Pi Pico, Raspberry Pi Pico W, Custom RP2040 board and the RAKWireless RAK11310

keever50 avatar Apr 05 '25 11:04 keever50

This is still an issue and blocks my developing the sx126x driver api port. I must get a different board because the RP2040 is unusable in this state. My old boards with a year old firmware still works perfectly and I cannot update them with the newest firmware because they will suffer with the same issues.

Meaning currently my only work-around is to use an old nuttx version.

Turns out that sometimes USB itself is not causing this problem. It can randomly show up with different configurations.

keever50 avatar Apr 13 '25 11:04 keever50

"Runs fine on my Framework laptop 13. Windows and Linux. (How strange)" also is too unstable now. I have none of my boards working anymore at any location, any configuration, any hardware.

keever50 avatar Apr 13 '25 11:04 keever50

I was not able to replicate this using the Raspberry Pi Pico W, the watchdog test starts and ends normally:

user_main: wdog test
wdog_test start...
wdog_test end...

End of test memory usage:
VARIABLE  BEFORE   AFTER
======== ======== ========
arena       3f950    3f950
ordblks         2        3
mxordblk    35f98    35f98
uordblks     78e8     7b88
fordblks    38068    37dc8

Is this issue intermittent? I tested on d976c66edfbcb26312ea8a841f4ffa191f53d769. USB connected (including datalines) and ran an OSTest through USB serial terminal.

linguini1 avatar Apr 17 '25 18:04 linguini1

Thank you for testing. Can you verify whether it works with different apps? How does the serial output look when running "help" or other apps? Because for me sometimes ostest also passes. Can you also try to flash it with a different USB port or machine?

I have the feeling there is something wrong with the clock configuration, which makes it run just about on the edge of failing. It could also be somewhere else.

Op do 17 apr 2025 om 20:49 schreef Matteo Golin @.***>:

I was not able to replicate this using the Raspberry Pi Pico W, the watchdog test starts and ends normally:

user_main: wdog test wdog_test start... wdog_test end...

End of test memory usage: VARIABLE BEFORE AFTER ======== ======== ======== arena 3f950 3f950 ordblks 2 3 mxordblk 35f98 35f98 uordblks 78e8 7b88 fordblks 38068 37dc8

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2813769050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKDUTZEIQLBATM5TNED2Z7ZSHAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJTG43DSMBVGA . You are receiving this because you authored the thread.Message ID: @.***> linguini1 left a comment (apache/nuttx#16139) https://github.com/apache/nuttx/issues/16139#issuecomment-2813769050

I was not able to replicate this using the Raspberry Pi Pico W, the watchdog test starts and ends normally:

user_main: wdog test wdog_test start... wdog_test end...

End of test memory usage: VARIABLE BEFORE AFTER ======== ======== ======== arena 3f950 3f950 ordblks 2 3 mxordblk 35f98 35f98 uordblks 78e8 7b88 fordblks 38068 37dc8

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2813769050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKDUTZEIQLBATM5TNED2Z7ZSHAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJTG43DSMBVGA . You are receiving this because you authored the thread.Message ID: @.***>

keever50 avatar Apr 17 '25 20:04 keever50

Thank you for testing. Can you verify whether it works with different apps? How does the serial output look when running "help" or other apps? Because for me sometimes ostest also passes. Can you also try to flash it with a different USB port or machine?

One thing I will say is that for as long as I've used NuttX on the RP2040, the serial output over USB has been finicky at best. The help command definitely has chopped output at least 1/2 or 1/3 times I run it, as with any other serial output. I also notice whenever I use the Python miniterm library as a serial console, there is strange characters in the output (rendered as something with a 'K' on my terminal). So that is definitely an issue for sure.

I will try some more OStests, using my XIAO RP2040 as well, and let you know.

linguini1 avatar Apr 17 '25 22:04 linguini1

I indeed have the same symptoms. This could be a seperate problem or related to memory corruption that got unnoticed until now.

I do not trust the chopped up output on both USB and UART. You can see this happening in some driver PR screenshots I submitted too.

Thank you for confirming that you also have the chopped up issue. That's very helpful to confirm it's not just me.

Op vr 18 apr. 2025 00:19 schreef Matteo Golin @.***>:

Thank you for testing. Can you verify whether it works with different apps? How does the serial output look when running "help" or other apps? Because for me sometimes ostest also passes. Can you also try to flash it with a different USB port or machine?

One thing I will say is that for as long as I've used NuttX on the RP2040, the serial output over USB has been finicky at best. The help command definitely has chopped output at least 1/2 or 1/3 times I run it, as with any other serial output. I also notice whenever I use the Python miniterm library as a serial console, there is strange characters in the output (rendered as something with a 'K' on my terminal). So that is definitely an issue for sure.

I will try some more OStests, using my XIAO RP2040 as well, and let you know.

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2814123294, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKGI6KBAYITJ7ZI2YSD22ASFZAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJUGEZDGMRZGQ . You are receiving this because you authored the thread.Message ID: @.***> linguini1 left a comment (apache/nuttx#16139) https://github.com/apache/nuttx/issues/16139#issuecomment-2814123294

Thank you for testing. Can you verify whether it works with different apps? How does the serial output look when running "help" or other apps? Because for me sometimes ostest also passes. Can you also try to flash it with a different USB port or machine?

One thing I will say is that for as long as I've used NuttX on the RP2040, the serial output over USB has been finicky at best. The help command definitely has chopped output at least 1/2 or 1/3 times I run it, as with any other serial output. I also notice whenever I use the Python miniterm library as a serial console, there is strange characters in the output (rendered as something with a 'K' on my terminal). So that is definitely an issue for sure.

I will try some more OStests, using my XIAO RP2040 as well, and let you know.

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2814123294, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKGI6KBAYITJ7ZI2YSD22ASFZAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJUGEZDGMRZGQ . You are receiving this because you authored the thread.Message ID: @.***>

keever50 avatar Apr 18 '25 09:04 keever50

I ran the OSTest on my XIAO RP2040 board and haven't had any issues there either (watchdog passes fine). Ran it twice in a row to be sure, still using the usbnsh config and reading shell from minicom.

linguini1 avatar Apr 18 '25 20:04 linguini1

Thank you,

The weird part is sometimes ostest actually passes for me. But when you try to do work, it returns. A heisenbug.

For now I am unfortunately unable to use nuttx which is a shame. I really hope someone can figure out this problem.

Also, the messed up serial output does not give me much confidence either.

I hope people can do more testing.

Op vr 18 apr 2025 om 22:12 schreef Matteo Golin @.***>:

I ran the OSTest on my XIAO RP2040 board and haven't had any issues there either (watchdog passes fine). Ran it twice in a row to be sure, still using the usbnsh config and reading shell from minicom.

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2816124166, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKGHNVH4N2P3NYEMWKD22FMEHAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJWGEZDIMJWGY . You are receiving this because you authored the thread.Message ID: @.***> linguini1 left a comment (apache/nuttx#16139) https://github.com/apache/nuttx/issues/16139#issuecomment-2816124166

I ran the OSTest on my XIAO RP2040 board and haven't had any issues there either (watchdog passes fine). Ran it twice in a row to be sure, still using the usbnsh config and reading shell from minicom.

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/16139#issuecomment-2816124166, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3DKGHNVH4N2P3NYEMWKD22FMEHAVCNFSM6AAAAAB2QM6CJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJWGEZDIMJWGY . You are receiving this because you authored the thread.Message ID: @.***>

keever50 avatar Apr 20 '25 08:04 keever50