Feature request: Add support for IP camera to capture images and/or video of the aircraft within range

[Romeo-Golf]

While Twitter Bot's may not be for everyone, this element stood out as being something which may have wider appear so figured it was worth starting a thread to discuss this and the future potential to use Open CV etc..

Perhaps as a first step could we consider a generic IP cam which lacks Pan, Tilt or Zoom... but has a fixed view of the sky...

First we would need to enter the LAT/LON and ideally Elevation of the camera, perhaps with the user locating them on a map if there was ever a web interface, or via the CLI using similar code to the ADS-B feeder scripts on adbs-receiver.

The user could then confirm the field of view, perhaps with a default value of 90 degrees.. and possibly also with the option to set different values for horizontal and vertical.

Orientation may be something that is less easy for the user to figure out without being able to see it overlaid on a map... but hopefully not to much of an issue for an advanced user to make a reasonable estimation of and enter via the CLI..

And there is probably similar considerations around the cameras elevation but perhaps like the field of view we can assume its aimed at the horizon and/or horizontally ;)

Lastly for future calculations we probably want to consider some kind of range value, probably only a couple of miles, to prevent it trying to take pictures of planes that are to far away to even register as a single pixel on a moderately priced IP camera.

So although that sounds like a lot to figure out, once all that information has been gathered its presumably relatively straightforward to use the locally gathered ADS-B/Mode S data to determine when there is likely to be an aircraft within the camera's field of vision? ;)

[kevinabrandon]

Adding a camera is exactly what I want to do next.

I'm pretty sure I'll be getting an IP based surveillance system for Christmas, and I plan on provisioning one of the IP cameras for just this application.

Like you pointed out, it wouldn't be very hard to add a rudimentary photo/video taker.

I intentionally added the angle calculations (azimuth and elevation), to the flightdata.py script because they'll be needed for this. The hardest part for a layman user will be measuring the direction their camera is pointing. They'll need to know both the compass, and elevation angles, along with position and height like you said. But once that is known it will be trivial to know if the aircraft is in frame or not. Then we'd have it take photos or video of any airplane within the minimum range. It should be very simple. For this OpenCV would only be used to overlay text about who or what the airplane is (according to the ADS-B data).

Later, I'd experiment with creating an OpenCV airplane detector/tracker which would use the video to detect the airplane. It would draw a nice little box or circle around the airplane as it's being tracked. Also if a pan-tilt-zoom camera is being used, the opencv tracker should give a smoother track, and allow for fuller zooming.

My final goal is to have it auto-upload video of every airplane that flys overhead to YouTube.

[Romeo-Golf]

Sounds good and I wish Santa brought me useful goodies instead of socks and slippers ;)

Once its at the stage of taking photos when it expects an aircraft to be in frame, based on where its told the camera is orientated, and able to overlay text data would it be possible to display where the code thinks the aircraft should have been?

Thus, without even having OpenCV detect the aircraft, it should be possible to use trial and error to tweak the less easy values like orientation and elevation angle until the expected position is over-layed exactly on top of the aircraft :)

If OpenCV is able to detect the aircraft, which might actually be easier at night with red and green lights hopefully standing out against a clear sky, then in addition to visual post-processing then perhaps it could even be possible to have the system auto-tune...

At least it should be possible with a camera that is pointed within 45 degrees of where its told its orientated, although probably nothing stopping it from iterating through the compass until it does start detecting objects with trajectories which match known ADS-B/Mode-S tracks ;)

[kevinabrandon]

Yeah, drawing where it thinks it should be, wouldn't be too hard.

I like the idea of it auto-calibrating using the video tracker.

[Romeo-Golf]

Thought these might relevant:

http://znah.net/plane-tracking-with-python-opencv.html

https://www.ethz.ch/content/dam/ethz/special-interest/baug/igp/photogrammetry-remote-sensing-dam/documents/pdf/Student_Theses/MA_Neff_FS2016.pdf

[Romeo-Golf]

Variable wise I'm guess it would need:

Camera_Latitude Camera_Longitude Camera_Altitude

These could potentially be inferred from dump1090, but worth considering the camera may not be in the exact same place as the receiver/antenna?

Camera_Bearing: 0 to 359 degrees Camera_Support_Pan: true/false

Camera_Elavation_Angle: -180 to 180 Camera_Support_Tilt: true/false

Camera_Vertical_FoV: assume 90 degrees? Camera_Horizontal_FoV: may not be same as vertical if wide-screen etc

Camera_Support_Zoom: true/false Camera_Maximum_Zoom: assume 1?

Anything else? :)

[Romeo-Golf]

Oh and Camera_URL, which could hopefully then be used to determine Camera_Resolution

[Romeo-Golf]

In addition to supporting IP cameras is it worth considering support for the PiCam and potentially the pan/tilt hat?

https://www.adafruit.com/product/1967

[kevinabrandon]

I like the idea of adding picam support.

Also the pan/tilt hat is a good idea too. I'm wondering if there should be intermediate interface for supporting pan/tilt/zoom. Then if someone has their own ptz option that isn't standardized, they could write a plugin themselves that would join the interface between this project and whatever ptz device they have. That way I (or any other developers) won't have to buy a plethora of devices for testing. And if someone has a nonstandard ptz there would be a possibility of them getting it to work if they are willing to write their own plugin.

It probably makes sense to do that for cameras as well. I could write a plugins for my ip camera and the picam, and then if someone else has a camera that is different, they could write a plugin to support it.

[Romeo-Golf]

yeah sounds like a good idea to modularize/pluginize both the camera itself and the PTZ.. and the PiCam seems like a nice and cheap way for folks to get involved, did Santa bring the IP cameras you were expecting?

One other camera type that might be worth considering is using gphoto2 with a standalone digital camera connected via usb, this may open up the potential to connect DSLR type cameras with zoom lenses ;)

[kevinabrandon]

I did get a few ip cameras, and I'm planning to set one up pointing at the sky very soon, no PTZ though :)

I just created a new branch called simple-cam where I'm going to assume a stationary camera.

I'm going to start with a simple GUI program that shows the live video, and then overlays a circle around any airplane flying in frame.

In the grand scheme of things this GUI will be used for calibrating the camera's orientation. There will be two opencv trackbars, one for it's heading angle, one for it's elevation angle. As you adjust these trackbars you will see the changes in real time, so you can fine tune things as the airplane is flying overhead.

So far I just added a bunch of items to the configuration file, most of them are what you suggested above. And then I added a simple gui that displays video from the cam_url. Hopefully in the next few days I'll get my camera set up and I'll be motivated to finish this first step. :)

[Romeo-Golf]

Sounds good and probbly a wise idea to start with stationary cameras only...

Im happy to help testing and have a clear 60-70 degree FOV on the approach to a major airport with ~1200 ADS-B hit per day..

And while I dont have a PiCam (yet), I do have a USB webcam and an IP cam (which servers up an MJPEG via HTTP) so could point both of those at the sky... albeit behind a window for now :)

One caveat is that I dont have a windows machine so if not already I would need the calibration GUI to be web based.. is that feasible?

Speaking of caveats I dont plane to use the twitter element so it would be great if the output could be made modular/plugin based such that it could generate an email instead of or in addition to a twitter post... does that seem like a useful addition?

[Romeo-Golf]

PS I tested my point and shoot Canon with ghpoto2 and sadly it doesnt support image capture; however my EOS DSLR does...

Which with a decent zoom lens I have been able to get good enough shots to read the tail number up to a couple of thousand feet, so although defiantly a longer term goal its worth considering that with gphoto2 driving a DSLR and a suitably calibrated PTZ mount it may be possible to capture images where the tail number could be OCR'd to provider absolute confirmation that the aircraft matches the ADS-B data!

[kevinabrandon]

The current setup will work with a usb webcam. Simply set cam_url=0 and opencv will simply grab the first usb webcam it finds. If you have two usb cams plugged in, you can set cam_url=1 to get the other one. It doesn't yet support the picam, but I think I'll add that next, it will probably be a "special url", something like cam_url=picam.

The GUI will be useful for calibration, but it is mostly for my own development purposes. I think for now, the goal of this simple-cam branch will be to create a script similar to the twitter tracker.py that will activate the camera whenever an airplane flies into a predefined alarm zone and then take a video, or a single photo when the airplane is closest to the center of the frame. Then it would do something like tweet, save to disk, upload to youtube or email it (all TBD).

The idea of getting tail numbers is essentially the projects end goal, but it may be awhile. :)

[kevinabrandon]

Here's a little video of my first attempt at tracking with the camera using only the ADSB information.

At the beginning of the video I adjust the trackbars in an attempt to set the camera calibration. You can see that the airplane is inside of the drawn circle for a short time after that. Pretty quickly as it comes closer the airplane and the drawn circle begin to diverge and eventually they go way off. At the end of this video, I do one last crazy attempt to reset the camera calibration and fail miserably.

I've got a few ideas as to why it is so bad at the end.

1. Orientation/position of the camera. The angles that the camera is pointing, and the position and altitude of the camera are probably off, and as the airplane gets closer it becomes more obvious.
2. Fisheye effects. I am not doing anything to correct for fisheye effects, and my camera has a fairly wide angle lens.
3. Rotation of the camera. I account for pitch (el) and yaw (az), but I don't account for roll. When the camera is pointing up at the sky it's hard to make it level.
4. Lastly, it could be that the ADSB information is lagging or inaccurate, but I do not think that is the case.

All these reasons point to moving away from trying to get perfect camera calibration, but instead make a simple video object tracker that would video track the target and then use the ADSB information to guess what airplane it is.

[Romeo-Golf]

Interesting watching and im sure a lot better than my first attempt would be ;)

Some considerations, in not particular order:

• Its been a long time since ive had to consider such math, but I would imagine that parallax gets interesting when the object your trying to track is heading towards you at 300 feet per second.. for you next test would it be possible to orientate the camera such that its perpendicular to the flight path?

• Also do you think the detection might have an easier time if you were able to repeat on a day/time with clear skies?

• I think that any user supplied values for orientation/position/rotation/fov may need to be considered approximate and, while unlikely to be easy, I suspect that automated calibration would be required before Pan/Tilt/Zoom is viable...

• But at this stage thats probably a long way off so the interim step could potentially be user assisted calibration whereby even if the user has to tweak the values, it should be possible to confirm that the camera position matches with a bit of ingenuity...

Some ideas to assist with that are:

• Plot the flight path of the nearest aircraft over the camera stream, thus potentially allowing calibration be either updating the configured values or even moving the camera...

• Try to calibrate after dark, where assuming there are any flights overhead it could be easier to track the aircraft nav lights flashing against a black backdrop..

• Or going one step further, if the sky is clear and not too polluted then it should be possible see the stars and thus follow the age old tradition of using them for navigation..

ie if we can use what we assume is a relatively accurate lat/lon and approximate altitude, then im sure its possible to figure out where the bright lights int he sky are expected to be and if this doesnt match what the camera see's then figure out what the difference is ;)

Hopefully further food for thought...

[Romeo-Golf]

For extra fun I took some screenshots of my camera which is approximately orientated at 255 degrees from north and with similarly approximate elevation angle of 20 degrees above the horizon...

http://imgur.com/a/NzKNg

Sorry there is only a couple of shorts with discernible planes in them but i found even the contrails useful, for example in the first image where the nice clear trail belongs to the craft at 35k ft on heading 70, ie coming from heading 250, is close enough to 5 degrees off the vertical axis to confirm the camera orientation is at least within 5 degrees of the 255 degree value I guesstimated..

Similarly I picked that frame as the aircraft is nearly out of the top of shot, so based on the ADS-B values which give an altitude of 10668 m and distance of 13.3 km gives an angle of 38.8 degrees, or near enough 40 degrees for the top of frame..

Whereas the second frame where the aircraft is a little below the middle of the frame, with altitude 6393 and distance of 15700, gives an elevation angle of 22.2 degrees...

And the third frame, where the aircraft hiding behind the tree about a quarter of the way up the frame is on final approach with altitude of 1250 m and 6500 m distance, this gives an angle of 10.8 degrees.

So although a gross over simplification and completely ignoring that few, if any, lenses are likely to give an image that is free of distortion... it should at least be possible to approximate the orientation and FoV for both horizontal and vertical aspects... but will likely require significantly better math and coding skills than I posses ;)

[Romeo-Golf]

Managed to get a couple of night shots, both were about 3 miles out and descending to 4k ft:

http://imgur.com/a/mDwWD

Note on the second image the light towards the lower left hand corner of the image is actually Venus which appears to flicker as its visible through tree branches, but does suggest that calibration from celestial bodies should be do-able even if only under certain conditions :)

[wiseman]

I've been interested in automatically capturing high-zoom images of overhead aircraft for a while. A couple links that might be relevant:

1. OpticTracker uses visual tracking to automatically follow aircraft using motorized telescopes. It can use ADS-B info to slew to the general area of sky. Demo video.

2. This isn't quite the same thing, but @kevinabrandon's video above reminded me of it: I did some experimenting overlaying ADS-B info on a live camera feed, only in my case the camera was mounted on a drone (that could sense its orientation and heading). I wrote a blog post about it, and here's a short demo video. Looking over my old code, I see that I also didn't try to handle roll or camera distortion :(.

[5foot8]

Here's a little video of my first attempt at tracking with the camera using only the ADSB information.

At the beginning of the video I adjust the trackbars in an attempt to set the camera calibration. You can see that the airplane is inside of the drawn circle for a short time after that. Pretty quickly as it comes closer the airplane and the drawn circle begin to diverge and eventually they go way off. At the end of this video, I do one last crazy attempt to reset the camera calibration and fail miserably.

I've got a few ideas as to why it is so bad at the end.

1. Orientation/position of the camera. The angles that the camera is pointing, and the position and altitude of the camera are probably off, and as the airplane gets closer it becomes more obvious.
2. Fisheye effects. I am not doing anything to correct for fisheye effects, and my camera has a fairly wide angle lens.
3. Rotation of the camera. I account for pitch (el) and yaw (az), but I don't account for roll. When the camera is pointing up at the sky it's hard to make it level.
4. Lastly, it could be that the ADSB information is lagging or inaccurate, but I do not think that is the case.

All these reasons point to moving away from trying to get perfect camera calibration, but instead make a simple video object tracker that would video track the target and then use the ADSB information to guess what airplane it is.

If you are still actively looking at moving this forward, you should definitely have a look at https://github.com/cnelson/maho. Mr Nelson has done a fantastic job of doing the aircraft tracking with PTZ camera. I worked on a version to accommodate my own requirements(maho-contrail), at the moment I want to add flightpath overlay, flight details and a method of lowering the highest altitude dependent on the local aerodrome cloud forecast, no point looking for planes above 30,000 ft if there is a solid layer of gray miserable cloud. I'd be very interested in the code used just to overlay the data, as in your demo YouTube video.

[Romeo-Golf]

Funnily enough I was thinking about this again recently after having stumbled across this article:

https://medium.com/@arunvenkats/automating-the-capture-of-airplane-pictures-with-raspberry-pis-ads-b-and-iot-software-39e25ddcf3ea

Interesting to see how things have come along since the first attempt at this I saw 5 years ago:

https://web.archive.org/web/20180129150726/http://simonaubury.com/the-pi-plane-project-whole-write-up/