Neural Control Mode

[SindreMHegre]

Solved Problem

Using neural networks for various tasks on UAVs in research is becoming more and more commonplace. My aim is to provide a module that can be used to replace the MC controller with NN to lower the barrier for using RL and NN on UAVs in general, and give researchers and hobbyist a place to start for NN use.

Solution

• Add TensorFlow Lite Micro as an inference lib into PX4
• Make custom board configs with neural controllers
• Make a module for NN control

Changelog Entry

For release notes:

Feature: Neural control mode for PX4, with possibility to use your own neural networks.
New parameters: MC_NN_EN, MIN_RPM, MAX_RPM, THRUST_COEFF
Documentation: read docs.px4.io/advanced/neural_networks, tflm and nn_module_utilities

Alternatives

We could also look into using Eigen or executorch

Test coverage

• Built and tested SITL with the x500 https://review.px4.io/plot_app?log=52502d02-009e-4d1f-baac-a49dd51cfdef (check neural_control topic)
• Built and tested HITL with pixracerpro and jMAVSim Hitl, note that the sim itself seemed a bit unstable and the network is not optimized for this model. https://review.px4.io/plot_app?log=43d2ba6c-3d05-4d7f-a31b-490a57b3e647
• Tested live on ARL LMF model: https://review.px4.io/plot_app?log=d101189d-672e-42a3-9d39-828c675dde57
• Tested live on the X500, but some problems on the platform caused a sub optimal test. https://review.px4.io/plot_app?log=eff5ef61-3318-4f74-ad82-bfa42388dbd5

Video and paper on the module. Master thesis coming later https://www.youtube.com/watch?v=lY1OKz_UOqM&t=1s https://arxiv.org/abs/2505.00432

Context

Other feature requests:

• Put in different controller parameters with possibilities to replace different parts: Full e2e, only control, only allocation, Allocation and control as two separate networks.
• Script to give thrust and delay parameters from a flight log

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-feb-19-2025/43827/2

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-feb-19-2025/43827/1

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/how-to-integrate-eigen-library-with-px4-firmware/44065/2

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/adding-an-external-flight-mode-from-the-fcu/44137/1

[Jaeyoung-Lim]

@SindreMHegre Why is the CI failing?

[SindreMHegre]

Why is the CI failing?

@Jaeyoung-Lim It has been failing ever since I rebased on main instead of v.1.15.3. But it's nuttx build errors, So I would guess that it's because of the toolchain change, specifically the line where I load the toolchain path with a local filepath.

I think it could be fixed PX4 switched from gcc-embedded 9.3.1 to 13.2.1 during bash ./Tools/setup/ubuntu.sh

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-mar-19-2025/44414/2

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-mar-19-2025/44414/1

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-mar-26-2025/44527/2

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-mar-26-2025/44527/1

[Jaeyoung-Lim]

I see a lot of reviews/comments here and there in this PR. I think we should try to get a first minimum MVP working and merged, rather than making everything too perfect as this is an experimental feature anyway

@SindreMHegre what are the blockers right now? I see CI is still failing, so I think this should definitely be addressed.

@MaEtUgR @Pedro-Roque @PetervdPerk-NXP any opinions on what a minimim MVP should be?

[SindreMHegre]

@SindreMHegre what are the blockers right now? I see CI is still failing, so I think this should definitely be addressed.

1. I agree with MVP, right now the biggest issue is that the CI fails, I just posted about it on discord, it seems like the submodule does not download the newest version of my fork. If i clone the repo with submodules it does not work, but if I cd src/lib/tflm/tflite_micro and then git pull origin px4_branch then it works. @Jaeyoung-Lim
2. There is one bug right now that the trajectory setpoint disappears after switching flight mode. I don't know exactly which config of the flight mode I should change for this, I made a default setpoint, but this is problematic if the drone is not close to that setpoint when switching. Discussed this with @MaEtUgR earlier.
3. I have a x500 here in Trondheim that I'm planning to test on. Right now I don't think the network will crash the drone, but I don't think it will fly very good either. Had some problems with ESC calibration that made the drone crash before getting to test neural mode. New parts should arrive any day, but I would like to have a live flight with the network on the x500 berfore merging

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-dev-call-may-14-2025-team-sync-and-community-q-a/45516/2

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-dev-call-may-14-2025-team-sync-and-community-q-a/45516/1

[mrpollo]

@SindreMHegre we finally merged #24832 and our CI and our builds should be passing with more than enough flash. Please rebase latest main!

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-dev-call-may-21-2025-team-sync-and-community-q-a/45657/3

[dagar]

I've forked https://github.com/PX4/tflite-micro to use for the submodule.

[SindreMHegre]

@dagar I tried removing the -nostdinc++ flag when building tflm, then I got a lot of errors with the nuttx include/cxx headers, so I tried removing them as well. Now the _U problem is back. How do I fix this without inserting my own version of ctype_base.h? Can something like the custom new be used? (Don't mind the SITL tests, as they are failing like 20% of the time and seems to be unrelated)

In file included from /usr/include/newlib/c++/13.2.1/bits/locale_facets.h:41, from /usr/include/newlib/c++/13.2.1/bits/basic_ios.h:37, from /usr/include/newlib/c++/13.2.1/ios:46, from /usr/include/newlib/c++/13.2.1/istream:40, from /usr/include/newlib/c++/13.2.1/sstream:40, from /usr/include/newlib/c++/13.2.1/complex:45, from ../../src/lib/tensorflow_lite_micro/tflite_micro/tensorflow/lite/kernels/kernel_util.cc:21: /usr/include/newlib/c++/13.2.1/arm-none-eabi/thumb/v7e-m+dp/hard/bits/ctype_base.h:44:35: error: '_U' was not declared in this scope 44 | static const mask upper = _U; | ^~ compilation terminated due to -Wfatal-errors.

[DronecodeBot]

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-2025-05-28/45749/2

[mrpollo]

@Jaeyoung-Lim can you please approve the PR so we can merge?

[SindreMHegre]

@Jaeyoung-Lim For reference, the files I've waited for dagar to check on is the small change to the GPS and GNSS files where i added a std:: and a static cast, I have not thoroughly tested those changes but they pass the CI at least. The second thing was the two files in src/lib/tensorflow_lite_micro/include. Those were added because of problems when using the C++ standard library. The fix file is for one of the header files that complained about those constants missing, so I defined them manually, don't know if that is the optimal way. The second is a header file that nuttx usually includes since the standard headers is not included, so I copy pasted the file. The only other suggestion I have is to copy paste the file at build time from the nuttx submodule, but if that is changed later then TFLM might break.

[Jaeyoung-Lim]

@mrpollo @SindreMHegre I do not feel confident that I have enough context regarding the major points raised by @dagar to approve the PR

[Jaeyoung-Lim]

The second is a header file that nuttx usually includes since the standard headers is not included, so I copy pasted the file.

@SindreMHegre What is the reason that you cannot include the file directly?

My main concern is that NuttX is Apache 2.0, so you would need to add relevant notices in order to be able to copy paste header files from the NuttX source.

[SindreMHegre]

@SindreMHegre What is the reason that you cannot include the file directly?

@Jaeyoung-Lim The include takes entire folders, I need just that file, if I include the entire folder it is located in then the other header files in the folder will cause the build to fail. I can however add a Cmake command to copy the file at build time, this way we don't include apache licenced files directly in PX4. And when we need the file, nuttx is included anyways, so it should not be a problem.

The biggest issue with this solution is that if someone changes the nuttx submodule then TFLM might break.

A third option, since we already have a fork of the TFLM repo is to copy the file into that fork instead?

Fouth option is to create a fix like I did for the constants, but I don't understand the error message well enough to make a fix like that.

The error I get without the header file is: In file included from ../../src/include/visibility.h:121, from : /usr/include/newlib/c++/13.2.1/cstdlib:173:11: error: 'system' has not been declared in '::' using ::system;

[Jaeyoung-Lim]

@SindreMHegre While I am not sure why it would be different to include folders and files jf you are using a specific header, you can include individual files as well.

The biggest issue with this solution is that if someone changes the nuttx submodule then TFLM might break.

This is the benefit. We should be able to detect this as soon as something breaks, rather than hiding it and disovering it until it is more expensive to fix (which is how it would be maintained). Copy pasting a header file can never be a better option in terms of maintainability, since this just multiplies the amount of maintenance effort that would be needed to ensure everything works.

[SindreMHegre]

@Jaeyoung-Lim, Now it uses the nuttx headerfile

[hamishwillee]

@SindreMHegre Thanks. I've run prettier and tidied the docs - fix line breaks, typos, highlighting of code. I haven't subedited the English itself, or checked that it makes sense.

The key here is that you can assume people know about PX4, but you shouldn't assume they know about neural networks. You'd assume basic information and provide links where possible.

I'll try have a proper look next week.

[SindreMHegre]

@Pedro-Roque

I built the SITL setup locally and upon activating the Neural Mode the vehicle did not respond to manual inputs. Is this module supposed to be used only with offboard control? I think it could be nice to have manual input control in the module so that users can test it out with QGC. Thoughts?

I think this mainly comes down to how the module is used and what the purpose is. The reason we disabled manual inputs are:

• The neural networks are experimental and does not provide a more stable controller than the standard one, so by giving manual inputs it will be even more unstable and it's hard to know if bad performance is because of a bad network or a bad pilot
• Benchmarking the module and network, and most research, is done by providing predetermined setpoints, so manual controls would not contribute here.

In the research we have done we have mainly looked at sim2real transfer, and the network is trained in sim with locked setpoints, so it is easier to compare the performance. And we have looked at trajectory tracking, and with predefined setpoints it is easier to get at comparable run in simulation and real world.

I have created a separate test module that feeds setpoints to the controller, which could be used in SITL, but I felt like it did not fit into PX4 as QGC has this functionality as missions. I could look into incorporating this in someway if desirable. The module can be found here: https://github.com/SindreMHegre/PX4-Autopilot-public/tree/main/src/modules/mc_nn_testing

This module is mainly meant to streamline the process of testing your own neural networks, not to be used as is or replace the standard controllers. One solution is to make a parameter for manual control in neural mode, but I don't see the relevance of being able to do this in the first place. I might have to update the documentation to make this more clear. I could also link to other resources like the paper or my masters thesis for those who want a deeper understanding: https://ntnu-arl.github.io/px4-nns/

As a last note, just wanted to say Great Work Sindre! We know it takes some effort to keep this going, but we are very grateful for the contribution!

Thank you! I'm also thankful for all your help in polishing the PR so the final version is as good as possible!

[Pedro-Roque]

@Pedro-Roque

I built the SITL setup locally and upon activating the Neural Mode the vehicle did not respond to manual inputs. Is this module supposed to be used only with offboard control? I think it could be nice to have manual input control in the module so that users can test it out with QGC. Thoughts?

I think this mainly comes down to how the module is used and what the purpose is. The reason we disabled manual inputs are:

• The neural networks are experimental and does not provide a more stable controller than the standard one, so by giving manual inputs it will be even more unstable and it's hard to know if bad performance is because of a bad network or a bad pilot
• Benchmarking the module and network, and most research, is done by providing predetermined setpoints, so manual controls would not contribute here.

In the research we have done we have mainly looked at sim2real transfer, and the network is trained in sim with locked setpoints, so it is easier to compare the performance. And we have looked at trajectory tracking, and with predefined setpoints it is easier to get at comparable run in simulation and real world.

I have created a separate test module that feeds setpoints to the controller, which could be used in SITL, but I felt like it did not fit into PX4 as QGC has this functionality as missions. I could look into incorporating this in someway if desirable. The module can be found here: https://github.com/SindreMHegre/PX4-Autopilot-public/tree/main/src/modules/mc_nn_testing

This module is mainly meant to streamline the process of testing your own neural networks, not to be used as is or replace the standard controllers. One solution is to make a parameter for manual control in neural mode, but I don't see the relevance of being able to do this in the first place. I might have to update the documentation to make this more clear. I could also link to other resources like the paper or my masters thesis for those who want a deeper understanding: https://ntnu-arl.github.io/px4-nns/

As a last note, just wanted to say Great Work Sindre! We know it takes some effort to keep this going, but we are very grateful for the contribution!

Thank you! I'm also thankful for all your help in polishing the PR so the final version is as good as possible!

Thanks for all the clarifications Sindre! So my suggestion would be:

• Allow the user to do position setpoints using the joysticks (so position control, but manual setpoints - this is what I meant in the original message but perhaps I was not very clear) or
• Provide repository / script that feeds offboard position setpoints to the onboard controller (so essentially what the mc_nn_testing is doing in your case, but not in the firmware level). You can take inspiration from https://github.com/Jaeyoung-Lim/px4-mpc/blob/master/px4_mpc/px4_mpc/mpc_quadrotor.py for the messages you need to send. Essentially, you need an OffboardControlMode with position set to true, and the position setpoint sent to the trajectory setpoint interface.

Good work!

[SindreMHegre]

@hamishwillee Thank you so much! I think it looks better now, did some small fixes as well. One diagram I made for my thesis that might be nice to include is something like this: I know it's not intended to use the ROS 2 API like this, but it does work and others might have use for the documentation. The figure was good to show exactly what I did, but I could update it to show what is optional, and add the optional topic I did not use. Do you think it adds some value or is the text enough?