New Gimbal Interface

[robertlong13]

Still a bit to do, but I have some stuff working. It's going to be a massive PR, so I want to get some of this out there to start review early.

• I have a test setup using RealFlight that I will need to document so anyone can play around with this.

• I have a lua script I need to publish that sends VIDEO_STREAM_INFORMATION messages so Planner can detect the URIs for the video.

• ArduPilot's CAMERA_FOV_STATUS message has a bug that causes the wrong units to be used for alt. Needs fixing.

• Most of the buttons and right-click options do not work yet

• Need a settings page so people can adjust the stuff like keybindings

• Things you can do

  • WASD to slew. Hold shift to go faster, ctrl to go slower.
  • Hold E to zoom in, hold Q to zoom out.
  • Clicking sets the POI to the lat/lon of the point you clicked, though more options are intended. (THIS ONLY WORKS IF YOU HAVE PATCHED CAMERA_FOV_STATUS to send the correct units)
  • Alt-F to take a picture
  • Alt-R to toggle recording
  • L to toggle between lock and follow
  • N to go neutral
  • H to point at home

(Terrible gif quality, had to really squeeze it down to fit in a github post, so there are a ton of artifacts; these obviously aren't in the real thing)

[rmackay9]

This is really great to see! Thanks for also finding the issue with FOV_STATUS. We added support ages ago but no GCS has used it yet so I guess we missed the bug.

[Davidsastresas]

Very neat! :)

[IuliuNovac]

How did you manage to get the simulation with a video feed up and running, is that gazebo?

[robertlong13]

How did you manage to get the simulation with a video feed up and running, is that gazebo?

I use RealFlight. I'm sure the same thing could be done in Gazebo, but I don't know my way around it nearly as well. You need this model and this lua script. You also need to be running on AP master (until the FOV message bug fix gets backported to 4.5)

I use a gstreamer pipeline to grab a cropped area of a monitor. I toyed with using OBS to stream as RTSP, but that introduced more than a second of latency.

dx9screencapsrc x=10 y=60 width=640 height=360 monitor=1 ! videoconvert ! video/x-raw,format=BGRA ! appsink name=outsink sync=false

Adjust the monitor, x, y, width, and height for your setup.

If you configure your autopilot to provide VIDEO_STREAM_INFORMATION messages, with gst://[pipeline] as the URI, this new UI can grab the pipeline from there automatically (it doesn't auto connect to it yet though; that's a planned feature).

The airport in this video was Pendleton Airport (KPDT), but I don't know if I can share my RealFlight model of that airport publicly. Instead, I'll share Base INFOCA, which has the side benefit of being within the real terrain model RealFlight pulled from Spain, so all the terrain actually lines up: Base INFOCA_AP.zip. The downside is that I went WAY overboard manually adding one metric crapload of trees to it. Disable trees with View > Scenery > Trees to disable them if your computer is choking on them. Launch SITL with -O 37.412620,-3.075734,1052.0,0

[robertlong13]

This PR is getting close to ready. All buttons and (almost) all keybindings are implemented, and I have a nice settings page where you can customize every keybinding.

Still needed:

• Click to track (need a testing setup for this one)
• Auto connect to first video stream if found
• Consider making a way to embed this on the flightdata page with minimap, and have the ability to swap between map with mini video and video with mini map.

[meee1]

need to tweak for android/apple - currently failing. system.drawing cant be used easerly....

[robertlong13]

Tracking added

[robertlong13]

Added new ways to display the new control

Full Sized. In this mode, there is a mini map by default, but it can be hidden with a button in the right-click menu.

Mini

Mini and full-sized can be swapped at any point with the "swap with map" when right clicking on the video.

And pop-out works exactly as before (except now it remains always on top, which I think is always better, you can always intentionally minimize it if it's in the way).

[rmackay9]

I've tested this quite a bit now and it works!

A few issues I found (and have already reported to Rob):

1. An error warning box some times appears if the camera is unplugged and then plugged back in again
2. right-mouse-button click on the video screen and select "Close" causes MP to crash

Some of the things I tested include:

1. confirming the VIDEO_STREAM_INFORMATION message's URL field is being consumed and used as the source
2. right-mouse-button-menu items are working (Retract, Neutral, Point Down, Yaw Lock, Take Picture, Start Recording, Stop Recording, Settings, Mini map, Swap with map)
3. Settings screen pops up and listed controls work (e.g. slew left, right, up, down). There were some that I could not test (like Zoom) because my camera doesn't support this

From my point of view, besides the two crash issues listed it looks very usable. I guess one enhancement (also already discussed with Rob) is it would be nice to reduce the number of clicks the user must do before the video appears. In particular the user must currently do this each time MP is started:

• Right-mouse-button-click on map and select "Gimbal Video", "Full Sized"
• Right-mouse-button-click on Video (which initially shows a big red X) and select "Video Stream"
• When the VideoStreamSelector window appears select from the "Detected Streams" drop-down and press "Connect"

[robertlong13]

• Implemented autoconnecting to the first reported stream
• Fixed up the GStreamer first-time setup. My setup accidentally relied on having put a video on your HUD at some point in time before you used this.
• Fixed a bug relating to properly handling an RTSP connection loss.
• Fixed a bug with the context menu options for swapping/closing the video control on the map.

[rmackay9]

I've retested this and it's working quite well. Some small issues that I found to do with the short-cut keys

1. the left click to pan/tilt seems to use the entire screen area instead of the smaller area where the video is shown. Perhaps it's related but I also found that clicking on the top of the video caused only a small change in pitch angle but clicking the bottom of the image caused a much larger change in pitch angle. I would expect that clicking at the very top or very bottom of the image would cause the same pitch change (but in the opposite direction of course)
2. slew up/down short-cut keys seemed reversed
3. retract short-cut didn't work
4. didn't re-request video-stream-info when re-connected (not a blocker)

Everything else seemed to work well. My testing included:

a. tested that all the menu commands worked (e.g. retract, neutral, point down, point home, take picture, start/stop recording, Mini map, swap with map, pop-out) b. tested all the short-cut keys (slew left, right, up down, zoom in/out, slew fast/slow, take picture, start/stop/toggle recording, lock/follow/toggle, neutral, point-down, home)

I think we should rebase this on master and mark as "Ready for review"

[robertlong13]

1. the left click to pan/tilt seems to use the entire screen area instead of the smaller area where the video is shown. Perhaps it's related but I also found that clicking on the top of the video caused only a small change in pitch angle but clicking the bottom of the image caused a much larger change in pitch angle. I would expect that clicking at the very top or very bottom of the image would cause the same pitch change (but in the opposite direction of course)

Oops, I'm pretty sure I'm explicitly rejecting clicks off-image, but must have messed something up. Good catch. Shouldn't be too hard to fix.

2. slew up/down short-cut keys seemed reversed

I agree that forward should pitch the camera forward (down) and back should pitch back (up), but I feel like everyone (even pilots) tell me I'm wrong and that up should slew up and down should slew down, which is ridiculous to me. I figured if I made the "correct" defaults here that I'd get complaints. If you think most people will actually expect "S" to tilt up and "W" to tilt down, I would be so very happy to change it to that.

3. retract short-cut didn't work

Just tested that the retract shortcut hit the same code path as the context menu. I don't have a key assigned to that one by defaut. Did you assign a key to it (that wasn't used by something else? I haven't implemented collision warnings and there is a kind of hidden priority to the keys if there is a collision; I should fix that)?

4. didn't re-request video-stream-info when re-connected (not a blocker)

It re-requests in OnMAVDetected, so if you disconnect in a way that Planner recognizes as a disconnection (hitting the button, or losing a TCP link), then it should re-request. It also requests it any time stream rates are re-requested, which is once every 8 seconds.

[rmackay9]

Hi @robertlong13,

Thanks for that. Re up/down, I think that "forward" on the keyboard (e.g. "w" is forward of "a" and "s") is "up" and that should make the gimbal pitch up.

Ok, re retract, I'm pretty sure I did have a keyboard shortcut conflict.

Txs for the info on re-connect. Perhaps I just didn't wait 8 seconds.

Thanks again! I hope @meee1 will have time to review and merge once he's happy with it

[robertlong13]

I think that "forward" on the keyboard (e.g. "w" is forward of "a" and "s") is "up" and that should make the gimbal pitch up.

Hrm, what you're describing is what it is supposed to do right now, unless I have my signs wrong in my mavlink commands (that must be it). Will test on my Siyi when I get home.

        public GimbalControlSettings()
        {
            SlewLeft = Keys.A;
            SlewRight = Keys.D;
            SlewUp = Keys.W;
            SlewDown = Keys.S;
            ZoomIn = Keys.E;
            ZoomOut = Keys.Q;

[robertlong13]

@rmackay9 I can't reproduce your reversal issue. Hitting up (W) definitely makes the gimbal pitch up by default. Can you confirm your keybindings? I confirmed that the keybinding defaults to what you're asking for, and I tested in SITL, and confirmed the MAVLink message going out matches the MAVLink spec. I didn't actually test with my Siyi, because I forgot how much of a pain it is for me to get it moving my current setup (though I should get that working again). If it's still going backwards for you, that makes me wonder if there's an AP implementation issue. Could you send a short tlog of you commanding slew-up (and only up, no down)?

[robertlong13]

@rmackay9, I found the issue with the click to pan. I was correctly clipping the mouse coordinates to the top and to the left of the image, but I was letting it overflow when clicking past the bottom or the right of the image. Clicking in the grey area on any side now should act just like hitting the edge exactly.

However, I couldn't reproduce exactly the symptom you drew in your picture. On the top side, even before my fix just now, clicking that red x did the same as the green x. It was only broken on bottom (which is why the bottom moved further than the top as you said in your message).

[rmackay9]

Hi @robertlong13,

The click-to-point seems great now, thanks!

.. and you are right about the reversal issue, the problem is with the Topotek gimbal driver! I've created a flight code PR to fix the driver (see https://github.com/ArduPilot/ardupilot/pull/29002)

So, all this all seems fine to me now, thanks!

[robertlong13]

@rmackay9, I've added warnings about binding clashes now on the settings page.

[rmackay9]

Thank you so much! This is a huge feature to get in!