iNavFlight
AHI/AHRS/Angle mode failure after about 1 to 3 minutes
Follow up on : https://github.com/iNavFlight/inav/issues/10943
Since I posted in this previous BR, I replaced the FC with the same make and model, and replaced the GPS, didn't change anything. Both FC worked fine on other planes, running under INAV 7.1, the chances that both these FCs are somehow faulty at the same time after being moved out of a plane is literally zero, I'd say.
Current Behavior
After a delay between 60 to 180 seconds flying in angle mode, the horizon goes bad, and within few seconds the plane is out of control and I have to switch to acro or manual mode to recover. I then have to fly long enough for the horizon to come back somehow, but it's never for very long, it stays more or less unstable until the end of the flight.
INAV v8.0.1, FC AtomRC F405 Navi, GPS 15+ satellites, plane is a forward swept 3D-printed flying wing of my own design, similar to say a Dart250 in size and performance.
In this flight the issue happens at 2:37 :
CLI Dump : INAV_8.0.1_DUMP.txt
CLI Diff : INAV_8.0.1_DIFF.txt
DVR : https://drive.google.com/file/d/1VXAqBhBZmMHfllpzaNu92wNJKTWBOjL1/view?usp=sharing
Raw log : LOG00062.TXT
Decoded log : LOG00062.01.csv
GPS trace : LOG00062.01.gps.zip
GPS :
Steps to Reproduce
I just have to fly the plane, it always happens, 100% so far over a dozen of flights, approximately 1 to 2.5 minutes after launch. I now have a sizable collection of about 10 DVRs of this happening. I'm not 100% ruling out vibrations, BUT, on a 3D printed plane these would be super easy to notice, and it does not appear to have any. Motor is clean and props are new.
For context, another DVR, trimmed down to the event : https://drive.google.com/file/d/1tzwkwAA5huQha9ohewhLkvsL319_37jk/view?usp=sharing
I have other similar planes running INAV 8.0.1 with identical settings (minor the hardware specific settings of course) and they do not do this ((Dolphin Pro with AtomRC F40 Navi Deluxe, Dolphin with Matek F405 Wing)
Expected behavior
Plane not diving to the ground to its mechanical death.
Side note, on the particular flight I choose as sample there was a LOT of wind, and it's a small plane so it was shaken quite bad, but the issue is the same on a calm day. But now I'm wondering if the stabilization being hard at work was the cause why it seemed to trigger later than all previous flights ?
Used to be at 60 to 100 seconds the previous days, and for this flight in really rough conditions it was at 165 secs. Also I enabled the glonass option so the sat count was higher than before, from 10-15 to 15-20.
Pretty sure this is caused by problems with the Z position estimations. A quick look at the DVRs shows numerous examples of position and Z velocities that contradict each other which I assume confuses the AHRS. Would need to check the logs for abrupt large changes in altitude/vertical speed that are likely responsible for the AHI suddenly losing alignment.
I'm thinking possibly a problem with Nav acc values given I noticed unrealistic values of up to 50g in the logs from #10943. The 50g didn't didn't remotely compare with the x. y. z g values from the accelerometer. Maybe something to do with the Nac acc gain calculations, something that's been changed/fixed in #10818 ?
Looking at the log above I don't see the abrupt changes in estimated altitude that were in the logs in #10943. The one thing that is obvious though is Acc Z is pretty noisy, flip flopping 0 to 2g constantly. Acc X and Y look OK in comparison. Unfortunately Nav Acc values weren't logged which makes it difficult to know exactly how acceleration is affecting the estimated positions.
I'll make another flight with the accelerometers logged, I forgot to enable this indeed. Maybe this evening, or tomorrow.
Ok so I harvested new datas !
Before that I change few settings :
Flight 81 :
The issue did NOT happen, to my surprise, first time in like 12 flights. But it was a short flight, 4 minutes (I'm vetting some old batteries) :
DVR : https://drive.google.com/file/d/1HbUYruHeon8ZGhn7iA23-FqyAQeo8EHK/view?usp=sharing Raw log : LOG00081.TXT Decoded log : LOG00081.01.zip
Flight 82 :
... so for the next I fitted a bigger lipo, and it happened, at 0:52, 2:32, 5:00, 5:50, 6:50
DVR : https://drive.google.com/file/d/1EaPJxyA8QU5IrdMx_AtdxGA_kyxDdXa8/view?usp=sharing Raw log : LOG00082.TXT Decoded log : LOG00082.01.zip
Looking at LOG00082 the problems seem to occur when estimated Vel Z and Pos Z start misbehaving possibly because Vel Z is noticeably higher than the GPS and Baro rates. e.g. around time 05.00 Vel Z looks overestimated by a facor of 2 and Vel Z and Pos Z contradict each other, i.e. Vel Z is -ve whilst Pos Z is increasing.
I misread the Log units for Nav Acc values, they're actually OK other than showing an excessive bias sometimes which would affect estimated Vel Z and Pos Z.
It might be useful to set debug_mode to VIBE and to check what posEstimator.imu.accWeightFactor is doing (VIBE debug 4) although looking at the log accVibe levels aren't significant so accWeightFactor probably remains at a value of 1 and therefore isn't affecting the estimate corrections. If that's the case problems with Nav acc values will just be caused by dubious posEstimator.imu.accelBias values being applied although it's hard to be sure given these values aren't logged. However, there are 3 unused VIBE debug fields so it would make sense to add the Nav accelBias x, y, z values here.
From the conversation it looks like a virbration or accelerometers problem. position estimators will be messed but position estimations will not have any affect on the AHRS.
Is the FC soft mouanted to the plane? Does the OSD shows 1.0G when placing the plane on different orientations?
Can we see a picture of the plane? Mechanical and aerodynamic effects could come into play.
This is the plane.
I havn't fully ruled out vibrations yet but I'd say it's very very unlikely that it could be the cause for several reasons :
- Vibrations are very easy to hear on 3D printed planes, and I hear none.
- It already went through 3 different motors (1407, 2204, 2306), 2 prop types, and 3 differents fuselages (prototyping phases ), and 2 different motor mounts, so the odds to it would vibrates each and every flight (about 15), over differents combos is quite low I'd say.
- Very low drag design, there's not much things "flapping in the wind", and the airfoil (PW51) is a well proven one, it does not have any known nasty effect or stuff like this.
- The bug always triggers after 1 to 2 minutes, it does not look like it's linked to high or low speed, or windy or calm days, high or low throttle etc
- FC has built-in soft mounts (AtomRC F405 navi), accels are calibrated and show all normal values.
Still, I cannot be sure 100% its not vibrations, but imho the odds are in the very low % numbers :)
https://www.youtube.com/watch?v=fl-8yHl3snQ
Yeah nothing obvious sticks out to me either. Shortly before the latest comment here, someone posted in the Discord a picture of their plane with two antennas on the nose each on 12" long stalks (30 cm). Something like that can certainly oscillate, so I like to get a quick look at a plane. :)
The built-in "soft mounts" is just solid columns covered by gummies on these FC. The bottom surface of the module is very solid to the imu unit. Try to use soft 5 mm-thick tape to mount the FC to the frame. 3D printed solid materials may be more sensitive to vibrations than soft form planes. The problemmatic vibrations can be small but very fast(high frequency), results in excessive acc/gyro reading
And I think it is not the cause of this issue, but it is better to stay away from the BMI270 gyro if AHRS reliability is your priority. Betaflight has a guideline to recommend against it https://github.com/betaflight/betaflight.com/pull/334
@OlivierC-FR Maybe a good question is: Is this plane (rifter mini) flying well with INAV 7.1? If so, then maybe the issue is with INAV 8. Have you tested it? I'm curious because I'm using 7.1 and planning to upgrade to 8.
@OlivierC-FR Maybe a good question is: Is this plane (rifter mini) flying well with INAV 7.1? If so, then maybe the issue is with INAV 8. Have you tested it? I'm curious because I'm using 7.1 and planning to upgrade to 8.
It's the exact same issue with INAV 7.1 ! I didn't test with INAV 6.