Actions list to improve NuttX quality and reliability

[acassis]

Is your feature request related to a problem? Please describe.

These are some of suggested actions we need to take to improve NuttX quality:

1. Automated Test and CI Integration (only will cover some corner cases)
2. Test Coverage Metrics
3. Expand and Improve Documentation
4. Standardize Board Port Quality
5. Better Unit Testing and Mocking
6. Stable API Guarantees
7. Regression Testing
8. Perform Benchmarks Metrics
9. Create Templates for Documentation
10. Static Analysis into CI
11. Create a base board with a standard MCU plug/insert connector

Describe the solution you'd like

This is a summary of actions we need to take to improve NuttX quality and reliability

Describe alternatives you've considered

No alternatives

Verification

• [x] I have verified before submitting the report.

[acassis]

1. Automated Test and CI Integration

It will help to detect for example if the board is not starting and if some testings (ostest, etc) is passing. There are many initiatives to have a Automated Test integrated with CI:

• Lup Yuen Test Bot and CI: https://lupyuen.org/articles/testbot2.html and https://lupyuen.codeberg.page/articles/ci.html
• Tomek NXDART (only proposal until now)
• Espressif NuttX Test Farm using Robot Framework
• Xiaomi Internal Test Farm (no info)
• Raspberrypi Test Farm (https://bitbucket.org/acassis/raspi-nuttx-farm/src/master/ not completely finished)

[acassis]

2. Test Coverage Metrics

Integrate code coverage tools like gcov/lcov for unit tests. Display and track code coverage over time to identify untested parts of the kernel, drivers, and libraries.

[acassis]

3. Expand and Improve Documentation

Improve Documentation/ to let end users to test boards easily. All boards should have basic instructions explaining how to install NuttX on it, currently almost none board has this basic instruction: i.e. https://nuttx.apache.org/docs/latest/platforms/arm/stm32f4/boards/stm32f4discovery/index.html We should enhance board-specific installation guides: How to connect the board (serial, JTAG, SWD). How to flash NuttX (dfu-util, OpenOCD, vendor tools, etc.). How to configure a simple project (make menuconfig, selecting board options). Add "Getting Started" tutorials for total beginners. Add troubleshooting sections for common problems.

[acassis]

4. Standardize Board Port Quality

Create a checklist for each board port to ensure minimum quality: Does ostest pass? Do basic drivers (UART, Timer, GPIO) work? Is SMP tested (if applicable)? Boards that don't meet the minimum criteria are marked as "experimental" or "unsupported".

[acassis]

5. Better Unit Testing and Mocking

Expand the apps/testing suite with more unit tests. Use frameworks like CMocka or extend the existing ostest usage. Mock drivers and hardware to allow kernel logic testing even without hardware.

[acassis]

6. Stable API Guarantees

Formalize API stability between releases (similar to "Stable API" policy in Linux kernel). Document which APIs are considered stable and which are still experimental. Add a deprecation process for removing/renaming public APIs.

[acassis]

7. Regression Testing

Maintain a regression test suite to ensure that previously fixed bugs do not come back. Basically when someone found an issue they should create a test to be integrated into ostest to detect it in the future. Set up automatic re-run of regression tests in CI when code is merged.

[acassis]

8. Perform Benchmarks Metrics

Create standard performance tests: Boot time benchmarks Context switch time Interrupt latency Track performance regressions automatically in CI.

[acassis]

9. Create Templates for Documentation

The idea is to have standard reference documents to be used when adding documentatio for a new Boards, a new Driver, etc

[acassis]

10. Static Analysis into CI

Using static analysis we can find simple mistakes that might have been introduced. Things like a function that forgot to return a value in some code path, or use of uninitialized variable, can be caught by static analysis.

[acassis]

11. Create a base board with a standard MCU plug/insert connector

The idea is to have a single base board with many peripherals (sensors, display, audio codec, ethernet, etc) and that board should use some standard MCU plug/insert board connector (like RPi CM4S or SparkFun MicroMod).

This way we can test existent peripheral drivers in many different microcontrollers. Also that base board (mainboard) could have features to facilitates HW tests.

[Jasinsky]

11. Create a base board with a standard MCU plug/insert connector

The idea is to have a single base board with many peripherals (sensors, display, audio codec, ethernet, etc) and that board should use some standard MCU plug/insert board connector (like RPi CM4S or SparkFun MicroMod).

This way we can test existent peripheral drivers in many different microcontrollers. Also that base board (mainboard) could have features to facilitates HW tests.

Hello, @acassis - just want to mention that products like that already exist - https://www.mikroe.com/fusion-for-arm#main-buy-field https://www.mikroe.com/development-boards/arm-32-bit

a bit pricey, but can save some hardware development time

[acassis]

Hi @Jasinsky thank you very much for that link!

Yes the idea is similar to that one, but in their case the mainboard just come with connector that user could plug many MicroBUS modules. My suggestion is to have a mainboard with all important components in place: Ethernet, LCD, Audio, Sensors, etc. And, of course, we could have some option connectors to support MicroBUS and PMOD modules.

Using their base board also could be a good starting point, but I think it could be more difficult to create a NuttX common board definition to work on their Fusion board because the users could move the modules of position and then the GPIO pins could be at incorrect.

[abisop]

3. Expand and Improve Documentation

Improve Documentation/ to let end users to test boards easily. All boards should have basic instructions explaining how to install NuttX on it, currently almost none board has this basic instruction: i.e. https://nuttx.apache.org/docs/latest/platforms/arm/stm32f4/boards/stm32f4discovery/index.html We should enhance board-specific installation guides: How to connect the board (serial, JTAG, SWD). How to flash NuttX (dfu-util, OpenOCD, vendor tools, etc.). How to configure a simple project (make menuconfig, selecting board options). Add "Getting Started" tutorials for total beginners. Add troubleshooting sections for common problems.

A crucial step: Please check : (https://github.com/apache/nuttx/issues/16432)

[linguini1]

3. Expand and Improve Documentation

To add to the list of action items, I think the NuttX website would benefit massively from featuring projects built with NuttX. Not just any example applications, but things like PX-4, industry companies' products, hobbyist projects beyond something super simple (i.e. not a blink LED project).

If we had any links to videos, or included some images of demonstrable projects it would be easier for new users to see how they can use NuttX for something real and robust.

[cederom]

1. Automated Test and CI Integration

Yup, I got distracted a bit by some important side-quests but slowly adding new boards to my local setup, upgrading hardware, etc, so internal stuff so far sorry but work in progress :-)

[NevynUK]

10. Static Analysis into CI

Using static analysis we can find simple mistakes that might have been introduced. Things like a function that forgot to return a value in some code path, or use of uninitialized variable, can be caught by static analysis.

Do you have an analyser in mind?

I started using cpp-check a few years ago and have it integrated into several ESP32 development pipelines using a GitHub action. Simple Dockerfile and a bash script coupled with an action.

I imagine a project the size of NuttX would complicate the script but it should not be be too difficult (famous last words).

I also added a clang-format checks to the same pipeline script.

The two together mean that any PR should have basic analysis and style checks performed before it gets to review.

[acassis]

Hi @NevynUK we didn't start it yet, but my initial idea was to took a look at coccinelle and frama-c, but yest cpp-check seems a nice option. Another option to take a look is SonarQube, the company behind it ran a test of NuttX source code and since it is open-source, maybe it could be possible to use it together with cpp-check for example.

I think the problem with clang-format is because it is not able to check the NuttX style correctly, some people already tried to adapt it in the past, but it wasn't able to catch all the coding styles.