Random 100% Thrust

[AlexKlimaj]

On recent main, occasionally I will get 100% thrust from the position controller. Same thing that happened last month testing the ekf height PR.

This is with multiple baros and a rangefinder.

Took off in position mode, then it went full thrust. https://review.px4.io/plot_app?log=f0acb7d2-7d1e-4ec9-92ca-0a816fd131f8

I was trying to land, sitting on the ground waiting for it to disarm. It wouldn't disarm so I throttled back up a bit to try again and it went 100% thrust. https://review.px4.io/plot_app?log=10ece277-09e1-4e67-be8e-ca7b87b7555e

[dagar]

It looks like it's coming straight from the position controller (not an estimator height reset, etc) when the hover thrust first becomes valid.

[AlexKlimaj]

@bresch It looks like it is a result of the new hover thrust math. https://github.com/PX4/PX4-Autopilot/commit/57fa9c545aefceef6adef82b454f72990c84e8a9

[bresch]

The compensation is correct but it seems that some other part isn't following it properly. I wasn't able to reproduce it in SITL; maybe the integrator saturates and as a result it cuts the compensation

[AlexKlimaj]

Another user reported a similar problem on discord. https://review.px4.io/plot_app?log=aefba76d-a6c4-486a-a61e-b7523d86d8fb

[AlexKlimaj]

@bresch @dagar I can reproduce it easily if I attempt to land, let the motors spool down, then start commanding a climb.

I adding some printouts for hover thrust valid and the integrator. It happens around 3:02. https://review.px4.io/plot_app?log=ca1149e4-4263-42cd-a634-ee671fb6db20

Looks like this PR caused it. https://github.com/PX4/PX4-Autopilot/pull/18971

[AlexKlimaj]

Just tried to replicate on main and I was unable to. I'll have to try again when the snow melts and I can test in the taller grass. https://review.px4.io/plot_app?log=590cca04-5afa-4d6e-8763-4a655dc0da26

[AlexKlimaj]

I have not been able to replicate this after the land detector rewrite. Closing for now.

[MaEtUgR]

@AlexKlimaj Thanks for the report. Could it be that https://github.com/PX4/PX4-Autopilot/pull/20581 solved this? It sounds very plausible specifically because you took off again which leaves room for exactly the problem I found.

[AlexKlimaj]

@AlexKlimaj Thanks for the report. Could it be that #20581 solved this? It sounds very plausible specifically because you took off again which leaves room for exactly the problem I found.

Yes that could be what fixed it too.

[TompsonTan]

I encounter this issue too, my firmware is based on v1.14beta1, log file https://logs.px4.io/plot_app?log=ec4160cd-0b5b-4204-8c87-0ed4168c3562

[TompsonTan]

[hendjoshsr71]

Hi all,

I've encountered this issue recently with a similar description as Alex's in how to reproduce it.

The branch is on a custom 1.13 with last merged in commits from main (6823cbc4140e29568f00e1211ae60e057adb1a1f) (Note of all my vehicles with this same firmware only one exhibits it to the extreme of shooting to the sky. Though you can audibly hear what may be this issues on others.)

[junwoo091400]

In the screenshot of the comment right above this, it seems like when the hover thrust estimate valid flag goes to true, the 80% thrust gets commanded. @MaEtUgR I guess you know the most about what's happening here?

[bresch]

Hi @hendjoshsr71 , could we get the log of that flight to investigate the issue please?

[MaEtUgR]

I know is that when the hover thrust gets valid this function is called to update the integrator: https://github.com/PX4/PX4-Autopilot/blob/9c001f2e40e366666aa8d654bc6f17ff51894b14/src/modules/mc_pos_control/PositionControl/PositionControl.cpp#L75-L88

I think we went through the math again and there's a test for it. What's left is either some kind of race condition or maybe if the integrator is already over its limit and the hover thrust corrects all the way and then the result gets limited again. The log would certainly help.

[hendjoshsr71]

Hi all,

Just to update you on the issue and a fix we have done on our aircraft. I was able to reproduce it in gazebo on our branch. We also put together a fix that alleviates part of the issue for us in gazebo and in our most problematic drone.

https://github.com/PX4/PX4-Autopilot/compare/release/1.13...hendjoshsr71:px4:pr/pos_control_max_thrust_bug

In PositionControl::updateHoverThrust(const float hover_thrust), _hover_thrust_new is not constrained by the same amount as the stored _hover thrust because we are pulling down on throttle and keeping ourselves stuck to the ground the estimate for hover thrust is much smaller than the 0.1 default constraint.

This then leads to a build up here

I threw in the other constraint as well to _vel_int(2) here just be more consistent as there are a lot of steps that use it before it eventually gets constrained here link

---

Probably the constraints (min/max) should be pulled in from MPC_THR_MIN and MPC_THR_MAX with checking for sanity not crazy small and not crazy large?

---

Really what "i think" should be done is some sort of smoothing on the acceleration or jerk in the poscontrol file. Partly as some "defensive" coding and partly so that these changes are less abrupt when hover_estimate.valid is true

Anyways I just wanted your thought to see if this seemed reasonable to do?

[hendjoshsr71]

Hi @hendjoshsr71 , could we get the log of that flight to investigate the issue please?

Unfortunately, due to customer constraints I can't share logs publicly. I can however produce most graphs as needed.

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-maintainers-call-april-04-2023/31415/1

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-maintainers-call-april-11-2023/31524/1

[mrpollo]

User @katzfey reports similar issues with ModalAI VOXL on v1.13-based firmware

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-maintainers-call-may-02-2023/31955/1

[junwoo091400]

I have done further testing with the main branch, with ATL Mantis Edu Quadcopter, and couldn't reproduce this issue.

I followed the following procedure:

1. Takeoff in Position mode
2. Hover (Let hover thrust estimator settle)
3. Land
4. Let the throttle in neutral position, or high, or low position, to a level where the vehicle stays on the ground

I observed:

1. With throttle low or in the middle, vehicle always diarms, as it detects being landed
2. With throttle slightly higher than middle, vehicle keeps idling / applying slight thrust on the ground, and doesn't exhibit weird throttle behavior

Here are the logs:

https://logs.px4.io/plot_app?log=60f24c0a-d422-472e-b5ff-2cea91ec4978

Other tests

https://logs.px4.io/plot_app?log=5fb34f03-6441-4156-aabe-d98502ed6d9f

[MaEtUgR]

Bottom line: This issue is reportedly reproducible on the 1.13 release but didn't happen later anymore most likely because https://github.com/PX4/PX4-Autopilot/pull/20581 changed the timing of takeoff detection. Still in theory any crazy hover throttle could have brought the multicopter position controller in trouble and that's why safety limitations specifically for the hover thrust update were added with https://github.com/PX4/PX4-Autopilot/pull/21512. So this is covered on 1.14.