[DO-NOT-MERGE] Mini 2 Support and waypoint optimizations

[m4xw]

Uses Frida to Support the Mini 2 by spoofing the DroneType as Mavic Air 2 and the Camera as Mavic Mini

Known Issues:

No Yaw limits for Gimbal, potential damage source
Simulator disconnect while flight makes the gimbal go crazy sometimes, unknown if this can cause potential damage. Persists on short restarts, on coldstart usually works a few times.
IMU shows abnormal, but no issues occur. Doesnt happen with Mavic Air 2 Camera spoofed
Mavic Mini Camera doesnt lets you set raw etc in the UX SDK, however its possible to set it in the DJI Fly app and it just keeps the values

Use on your own risk.

You will need to additionally have https://gist.github.com/m4xw/bcd66abb39f711399f7ff2e5b6be20e3 on your SDCard. The location is specified in gadget.config.so (fake shared lib, just a json)

Its inactive if the file isnt found. I am still not finished tuning it further and i cant recommend people using it widely yet. So bold disclaimer

Waypoint support is done by feeding it a csv in litchie format, i just crudely whipped that up now for metashape mission support.

Results have been very decent. Bare in mind this probably breaks a bunch of other use-cases, I just want to start raising some discussion on the topic!

Example CSV:

[m4xw]

Not a way to block a thread, you thrust the task preemtion to prevent you from a total block, a CPU waist. Better to use nano sleep as that force an emidiet task preemtion.

Well if you want to make it ideal, use condvars i guess but no clue how to do that in java. I waste cpu there deliberately as this executes on a dedicated thread that I spawn in MainActivity for my waypoints (its not good for calls outside of it). But I guess a small sleep wont hurt here to yield. Its a hack right now either way.

Also please ignore my lack of getters and setters for now ;)

[m4xw]

Overall I got the waypoints in a good state but I still struggle with the PID for a "cruising" mode. Currently flying a bunch of circles and the slowdown on approach pretty much ruins the footage. So i probably want to inject some better logic for the curvesize and tell it the next waypoint locations earlier and have the PID not slow down so much. Pretty much the opposite of how i want it for terrain surveys.

[m4xw]

https://www.youtube.com/watch?v=H1TvRwUAFy0 Its almost like yaw doesnt keep up (thats the POI mode btw, but without any active gimbal changes) Edit: Yea seems to be the yaw error margin

[The1only]

I will add this to main, thank you.

Best regard Terje Nilsen 9Tek AS Norway

On May 31, 2021, at 13:52, m4xw @.***> wrote:

 @m4xw commented on this pull request.

In app/src/main/java/sq/rogue/rosettadrone/DroneModel.java:

@@ -822,6 +835,11 @@ void send_autopilot_version() { msg_autopilot_version msg = new msg_autopilot_version(); msg.capabilities = MAV_PROTOCOL_CAPABILITY.MAV_PROTOCOL_CAPABILITY_COMMAND_INT; msg.capabilities |= MAV_PROTOCOL_CAPABILITY.MAV_PROTOCOL_CAPABILITY_MISSION_INT;

•    msg.os_sw_version = 0x040107;
  

•    msg.middleware_sw_version = 0x040107;
  

•    msg.flight_sw_version = 0x040107;
  

@The1only Just wanted to let you know that this silences the startup fw version error that occurs with QGroundControl (tho I dont think version is encoded right, but it does the trick..) Fwiw I am playing around with uGCS too and there it still chokes on receiving version info's until a timeout occurs, so it might be waiting for something else in this msg.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[m4xw]

On another note, just did 2 other testflights and seems my yaw issue comes from the curvesize early-handoff to the next waypoint introducing a noticeable yaw error that my code tries to correct too quickly. I might need to estimate the ideal yaw..

[m4xw]

Tried with high output filtering for the yaw/heading pid so it relies more on older data and that seems to get close for a decent POI solution and early handoff. The error range seems within 3-10m at 3m/s cruising between waypoints depending on orientation. https://www.youtube.com/watch?v=gF7S-Sv7Lcs Still need to dial it in a bit more and probably some math to approximate it smoother but getting there... Maybe its even right with higher distances / altitudes but I cant test that here anytime soon.. So finetuning it is.

[m4xw]

Just a meta comment to clarify the licensing situation: Frida's libgadget is licensed under a GPL/LGPL subset,so I will make use of its wxWindows Library Licence, Version 3.1 exception, which is basically LGPL. My fork should be considered LGPL for that matter (since its a not optional dependency for my fork and required to function with the intended goal of using the Mini 2)

Additionally my hook code should be considered under the Beerware License ( https://fedoraproject.org/wiki/Licensing/Beerware ).

Further, I will also make a exception for @The1only specifically, any contributions of mine that ends up in this repo to be considered under any license he sees fitting (BSD or whatever). In accordance to his will, other people may contribute my code as well under his terms. Consider this a CLA on my part.

[The1only]

Thank you for all you’re work. I will add some of it immediately and some I will have a closer look at this summer.

[m4xw]

Speaking of the devil, just pushed more changes. Also encountered a spooky flight controller crash that i mitigated. Not sure if only Mini 2 is affected. Happens when quickly stop start recording after having a few minutes of footage. Just syncing to the result doesnt mitigate it... Dont want to know what happens if it would be in flight lol

[m4xw]

I want to add for anyone trying POI: Theres a edge case when POI is right above the drone causing a error in the PID controller, so i guess i make a min absolute dist (jumps left n right in heading and never reaches the target)

[m4xw]

@The1only Critical fix in https://github.com/m4xw/rosettadrone_mini2/commit/f3371d1c9538064360e8a54e99c6d475ea05ea04 Super cursed Issue.. just for your notice. Same applies to the recording issue Also not every photo taken triggers a store apparently So we'd want to have it for recording as well..

[m4xw]

Waypoint interpreter is working nicely so far, bypassing all DJI limitations by design. I dont think the action logic was ever verified tho, QGC always seems to send a WP + follow up actions so they need to be assigned to the currentWP and update the object in the list. The waypoint xyz was absolute garbage (some negative -2XX fixed val) and NaN as Z so we cant add another waypoint for it and in the survey logic it then tries to make tons of intermediates to antartica or smth.

[m4xw]

I do wonder tho, heading is always 0, is it supposed to set the heading to the next wp by default, hmm.

[hodoje]

Is there a plan on finishing the support for DJI Mini 2?

[m4xw]

Is there a plan on finishing the support for DJI Mini 2?

Mini 2 support among many other things is finished since a while but its required to overhaul some stuff and revert several pieces of code to get this merge-able upstream. The alternate video decoding needs to be made a option too (latency vs reliability) and the DJI waypoint processor needs more love as well i need to fix the automatic intermediate waypoint generation for surveys. Bummer that litchi turned their planner private now (maybe i pissed them off?), so really need to improve wp planning next. Also there seems to be a issue with video feeder data rate which needs a manual action rn.

[hodoje]

Aha, no problem. Just wanted to know if it will be finished at some point. Thanks!

[kripper]

Hi @m4xw,

I took a look at your work. You probably invested a lot of time in testing different things but your code in general is like a hack. On the one hand, you have shown a lot of interest in contributing to the project, but on the other hand, it seems like you didn't really want your code to be merged upstream. Wouldn't you like to clean your code and see it merged to the upstream repo? At least the code that takes and processes the mavlink waypoints (not the Linchi stuff loaded from the sdcard). Once merged, your code will live and evolve forever. Otherwise, it will probably be vapor. If you want, I can help you to merge it upstream.

[m4xw]

Hi @m4xw,

I took a look at your work. You probably invested a lot of time in testing different things but your code in general is like a hack. On the one hand, you have shown a lot of interest in contributing to the project, but on the other hand, it seems like you didn't really want your code to be merged upstream. Wouldn't you like to clean your code and see it merged to the upstream repo? At least the code that takes and processes the mavlink waypoints (not the Linchi stuff loaded from the sdcard). Once merged, your code will live and evolve forever. Otherwise, it will probably be vapor. If you want, I can help you to merge it upstream.

Other than the original mini 2 support via frida which is no longer needed and maybe the statemachine changes, looking at you gotoNoPhoto, i dont consider it much of a hack, but yes its low effort on my end as I barely put time into it.

Video handling was rewritten to work on offscreen buffers, so stream can work when the app is minimized and smooth changes between map and video view, adds missing commands, Stream duplication, Waypoint interpreter and csv processors and adds basic stuff for occusync support

Anyway, I just work on it every other time i use my drone, which has been rarely as of late, so I dont have time to clean redundant or broken code, but it is technically still in active dev and its due for a rebase on my end. Also please remember that my license agreement only applies to the1only as of right now, so he must commit the change to be license compatible.

Generally i am not against upstreaming parts of the code, its just not ready yet and i plan to eventually overhaul a lot (and i still need to fix mission stuff like intermediate waypoints) and I plan to add telemetry logging for AI stuff etc

[m4xw]

Thanks for the offer tho, i will get back to it when the time has come

[kripper]

Hi @m4xw, please get back in sync with upstream master branch. Wayoint missions are now officialy supported for all DJI Mini series using the MissionManager class.

[m4xw]

Hi @m4xw, please get back in sync with upstream master branch. Wayoint missions are now officialy supported for all DJI Mini series using the MissionManager class.

I thought I made it clear that your changes dont fall under my CLA and are therefore license incompatible Nor do I appreciate my PID tunings, cruising mode etc being copied when you arent aware what they are there for without even giving credits.

Please follow the proper procedure (have the1only commit it or have him explicitly acknowledge it under the terms of my CLA, see my post, BEFORE merge) or explicitly ask for permission to relicense.

[m4xw]

Also your impl can make the flight controller crash midair (on mini 2). Edit: Nvm seems you backported that piece, missed that on first review

[kripper]

Hi @m4xw,

I read your post about Frida which was GPL and the licensing problems (we won't use Frida). My understanding was that you wanted to contribute, and also that you wanted and allowed Terje to merge your code to upstream which I did on his behalf. I didn't expect that your intentions were others. I also saw in your other posts that you were inviting people to take a look at your code to help them fix issues with the video.

I didn't want to upset you. Actually, I wanted you to be happy and keep you in sync with upstream.

I have been spending over 200 hours in the last weeks with almost no sleep to revive this project, bring the community together, close issues, fix bugs, add new features and share all my work with the community. I'm sorry if I was insensitive to you.

We are very thankfull and appreciate your contributions. We know hacking on Rosetta takes a lot of time, even when the results are only 3 lines of code.

Would you please authorize us to use your contrbiutions to Rosetta under the BSD license? If so, I will be happy to add your name in the code if you are happy with it. We would also be very happy if you continue contributing in the future by opening new pull requests or comments.

PS: Your repo is still published under the BSD license. Maybe you want to change this here:

[m4xw]

Generally i am not explicitly against it, but it was a legally binding statement from my part and only he may do it (or authorize it in written form, this cant be after the fact, legally speaking).

Would you please authorize us to use your contrbiutions to Rosetta under the BSD license? If so, I will be happy to add your name in the code if you are happy with it.

This isn't possible after it got merged. Theres no such thing as retroactive licensing. Doing that requires a settlement and that implies acknowledging the license violation.

Your repo is still published under the BSD license

You are correct that the visual indicator is wrong and I probably should explicitly say it on the repo itself, but due to the frida dependency it all becomes GPL on my repo, no matter what it says there.

I know I may sound picky here, but thats just the usual license mess.

So, anyway, i am just slightly salty because I explicitly stated that on several occasions.

So here's what we gonna do:

I declare a settlement of the violation on the condition that my copyright is added and we call it a day (from that point forward it will count as dual-licensed on upstream rosettadrone [this repo]). I generally don't have hard feelings, but if i declare a CLA, then the terms need to be upheld. Nothing against you but I dont give other persons randomly copyright over my code without some strings attached! I even considered the case that other people PR my work (which is this exact case) and all it needed is a one liner acknowledgement of the1only on the PR before merge ;)

[m4xw]

Also, to your commit. Slow mode == Cruising Mode, Quite literal meaning. Its cruising without a target destination due to early waypoint handoff and PID Controller Settings that cause it to do smooth movement and its for continuous actions (panning while recording or non-stop topdown imagery). This provides the smoothest movement but is vulnerable to Wind It will always over/undershoot the last target and undershoot intermediates.

Without it (>3m/s speed) the PID Controllers are fully active and try to combat over / undershooting and its most on-the-dot and ensures it will reach the destination with the given parameters. The PID Configuration values were experimentally determinated in Simulator and at medium to high wind speed in practice. There may be extra delay between reaching the position, confirming the destination and executing the action (check that it sticks near dest for a period)

Also quite sure u need to backport the earth size change or sim will fly curved lines on absolute movements (cant remember for sure anymore tho)

[kripper]

from that point forward it will count as dual-licensed on upstream rosettadrone [this repo]

I don't understand. Your are asking to change the BSD license of Rosetta?

[m4xw]

from that point forward it will count as dual-licensed on upstream rosettadrone [this repo]

I don't understand. Your are asking to change the BSD license of Rosetta?

No, just my code is BSD+GPL dual licensed on paper. You are free to have it BSD only upstream

[kripper]

Ok, thanks. Please provide me your full name or the verrbatim text you want me to add as a comment in the code (or wherever you want it to be placed).

[kripper]

Or maybe a pull request so you appear as a contributor on Github.

[m4xw]

Or maybe a pull request so you appear as a contributor on Github.

Will do, might as well double check the impl then and backport some other fixes of mine so probably wont do before next week or so, until then thats fine i guess