Make video_sample and other video commands 360° aware

[realchrisolin]

This is more of a feature request than a bug and I'll probably be doing a lot of the work for this myself, but I still wanted to create some sort of "ticket" for tracking purposes and upstream feedback.

Issue: Images derived from 360° videos using video_sample are not injected with the metadata needed so they're recognized as 360° images, resulting in bad segments when uploaded

The current solution I'm looking at is integrating https://www.github.com/google/spatial-media to detect if 360° metadata exists in video media and add the metadata to any images created from it. It's also designed to be worked with using Python2.7, but so far I haven't had any issues uses it on 3.7, so this shouldn't be a pain point if and when this tool is updated for Python3.7 compatibility.

[joshinils]

Still a problem, it seems.

I want to extract samples from a GoPro max video. Because the GoPro max single image time-lapse setting only allows a minimum time of 2 seconds, which for my liking is too long.

I need to manually stitch the video with the proprietary non-linux (i.e. windows) software from GoPro, then I can use that video (which has spherical metadata) to sample from.

So now that there is python 3 support on mapillary_tools, can this be implemented? Or do I need to manually inject the metadata into the sampled images to make them detect as spherical?

I only looked at the intermediary files while ffmpeg is still running, though.

[ptpt]

@joshinils Have you tried the direct uploading? The server will stitch the frames for you.

mapillary_tools process_and_upload gopro_max.mp4

[joshinils]

No, I need to correctly correlate the images to the gpx track. The timestamp metadata is wrong in either the video or gpx, I don't remember anymore when I recorded them.

But JOSM has the tooling to move the images along the gpx track and save the new location into the exif data, with a plugin.

Only after can I upload the images at their correct location.

Since I need to process the images with a script which I wrote for this purpose anyway, I manually inject the 360-metadata myself anyway; https://gist.github.com/joshinils/2ca6dd75ac6c1093b4d4fd2d60c68145

See my (German) OSM-diary entry; https://www.openstreetmap.org/user/cyton/diary/401670

[ptpt]

@joshinils Thanks for the context.

Curious why you use external GPS tracker. The GoPro Max does support GPS and it embeds GPS track in the video. See https://help.mapillary.com/hc/en-us/articles/360012674619-GoPro-MAX

[joshinils]

1. I did not assume a gps track was included in the video.
2. after stitching, it is gone from the output video,
3. so, only the unstitched *.360 video off the SD card has a gpx track, the stitched *.mov does not
4. I don't know how to extract the gpx track from the 360 file and include it in the mov file

When I do: --geotag_source "gopro_videos" --geotag_source_path GS015642.360

mapillary_tools video_process GS025642.mov --geotag_source "gopro_videos" --geotag_source_path GS015642.360 --video_sample_distance -1 --video_sample_interval 0.1 --skip_process_errors --interpolation_use_gpx_start_time

or --geotag_source "exif" --geotag_source_path GS015642.360

mapillary_tools video_process GS025642.mov --geotag_source "exif" --geotag_source_path GS015642.360 --video_sample_distance -1 --video_sample_interval 0.1 --skip_process_errors --interpolation_use_gpx_start_time

The images are not geotagged.

(But still, the images do not contain 360° metadata)

[joshinils]

Plainly feeding the unstitched .360 video file (which does include some gps metadata) does not work, the images are not geolocated;

mapillary_tools video_process GS015642.360 --geotag_source "exif" --video_sample_distance -1 --video_sample_interval 0.1 --skip_process_errors

nor

mapillary_tools video_process GS015642.360

inserts location data, or other metadata (like the accurate timestamp of when the frame was taken, with millisecond precision) allowing me to correlate after the fact with JOSM