Add camera tracking module

[srmainwaring]

Add a camera tracking module to support external camera tracking where the object tracker and gimbal controller run on an onboard companion computer.

The module is intended to work with Asif Khan's (@khanasif786) 2024 GSoC Visual Follow Me project and the modules make extensive use of the Python examples developed as part of the project.

Status

• Currently WIP, raised as PR to aid discussion and facilitate testing of the camera tracking support.

Related Blogs and PRs

• https://discuss.ardupilot.org/t/gsoc-2024-wrapping-up-visual-follow-me/123232
• https://github.com/ArduPilot/ardupilot/pull/27773
• https://github.com/ArduPilot/ardupilot/pull/27903

Details

There are two parts to the package: 1. a MAVProxy module camtrack that provides a GUI showing a RTSP video stream that allows a target to be selected, and 2. a Python script intended to run on a companion computer that receives camera tracking commands relayed from the flight-controller, attempts to track the object using a configurable tracking class, then updates the gimbal pitch and yaw rates.

The GUI module reuses the mp_image module by subclassing from the objects in that module so the tracking methods can be overridden. This is required to allow the tracking logic to be moved out of the GUI and into a separate module onboard_controller.py that may run either on a PC (in the case of simulation) or on a companion computer (in the case of a real vehicle).

The module will work with both simulated (Gazebo/SITL) and real vehicles and is tested with both.

Testing - Gazebo

A modified version of the omnirover_playpen.sdf world in ArduPilot/SITL_Models contains objects of interest and two models equipped with gimbal mounted cameras.

gz sim -v4 -r omnirover_playpen.sdf

Figure: Gazebo omnirover_playpen world for testing camera tracking

Enable the UDP stream

gz topic -t /world/playpen/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image/enable_streaming -m gz.msgs.Boolean -p "data: 1"

Convert to a RTSP stream using the scripts in ardupilot/libraries/AP_Camera/examples:

python gst_udp_to_rtsp.py

Launch a SITL session for the iris quad-copter (DDS support is optional):

sim_vehicle.py --debug -v ArduCopter -f gazebo-iris --model json --enable-dds --console

Load the module:

STABILIZE> module load camtrack

Run the onboard controller script:

python ./MAVProxy/modules/mavproxy_camtrack/onboard_controller.py --master 127.0.0.1:14550  --rtsp-server rtsp://127.0.0.1:8554/camera

Shift + Left-Mouse-Click to select a target. Ctlr + Left-Mouse-Click to cancel targeting.

Figure: MAVProxy camtrack module GUI

Figure: onboard_controller.py terminal output

Figure: MAVProxy console output