v0.7.3+ breaks MP4 files from Mio dashcams (without creation_time)

[littlebtc]

Basic information

• Release version: v0.7.3 & v0.7.4
• System: Windows 10 with WSL 2 Debian
• Capture Device: Mio 791

Steps to reproduce behavior

1. Provide a pair of MP4 video and NMEA file from Mio dashcams.
2. Running mapillary_tools with command like:

mapillary_tools video_process --video_import_path ~/todo/FILE210916-094020F.MP4 --advanced --geotag_source "nmea" --geotag_source_path ~/todo/FILE210916-094020F.NMEA --overwrite_all_EXIF_tags --use_gps_start_time

Expected behavior

The video should be able to sample and process.

Actual behavior

It will raise an KeyError

Traceback (most recent call last):
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/__main__.py", line 93, in main
    command.run(args)
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/commands/video_process.py", line 390, in run
    for k, v in vars_args.items()
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/process_video.py", line 106, in sample_video
    verbose,
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/process_video.py", line 148, in extract_frames
    video_start_time_obj = get_video_start_time(video_file)
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/process_video.py", line 219, in get_video_start_time
    video_end_time = get_video_end_time(video_file)
  File "/home/littlebtc/.cache/pypoetry/virtualenvs/mapillary-XjLv6BsC-py3.7/lib/python3.7/site-packages/mapillary_tools/process_video.py", line 202, in get_video_end_time
    time_string = FFProbe(video_file).video[0]["tags"]["creation_time"]
KeyError: 'creation_time'

Corresponding data

Output for ffprobe -loglevel quiet -show_streams -print_format json ~/todo/FILE210916-094020F.MP4:

{
    "streams": [
        {
            "index": 0,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "Baseline",
            "codec_type": "video",
            "codec_time_base": "1001/120000",
            "codec_tag_string": "avc1",
            "codec_tag": "0x31637661",
            "width": 1920,
            "height": 1080,
            "coded_width": 1920,
            "coded_height": 1088,
            "has_b_frames": 0,
            "pix_fmt": "yuv420p",
            "level": 40,
            "chroma_location": "left",
            "refs": 1,
            "is_avc": "true",
            "nal_length_size": "4",
            "r_frame_rate": "60000/1001",
            "avg_frame_rate": "60000/1001",
            "time_base": "1/60000",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 10819809,
            "duration": "180.330150",
            "bit_rate": "20000865",
            "bits_per_raw_sample": "8",
            "nb_frames": "10809",
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0
            },
            "tags": {
                "language": "eng"
            }
        },
        {
            "index": 1,
            "codec_name": "adpcm_ima_wav",
            "codec_long_name": "ADPCM IMA WAV",
            "codec_type": "audio",
            "codec_time_base": "1/32000",
            "codec_tag_string": "ms[0][17]",
            "codec_tag": "0x1100736d",
            "sample_fmt": "s16p",
            "sample_rate": "32000",
            "channels": 2,
            "bits_per_sample": 4,
            "r_frame_rate": "0/0",
            "avg_frame_rate": "0/0",
            "time_base": "1/32000",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 5770458,
            "duration": "180.326813",
            "bit_rate": "256000",
            "nb_frames": "5770458",
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0
            },
            "tags": {
                "language": "eng",
                "handler_name": "Sound Media Handler"
            }
        }
    ]
}

It seems that ffprobe cannot get the creation_time from videos produced with Mio Dashcam.

If you need a working example please tell me, I will upload one.

And it seems that #426 is the root reason of it?

Additional information

I verified it worked in v0.7.2 but not in either v0.7.3 and v0.7.4.

[ptpt]

@littlebtc Thanks.

The new release v0.8.0 should fix it https://github.com/mapillary/mapillary_tools/releases/tag/v0.8.0. v0.8.0 will use the gps start time instead of the video start time.

Try with v0.8.0:

mapillary_tools video_process ~/todo/FILE210916-094020F.MP4 --geotag_source "nmea" --geotag_source_path ~/todo/FILE210916-094020F.NMEA --overwrite_all_EXIF_tags

Let me know your feedback.

[littlebtc]

@ptpt This issue seems to be fixed, but I ran into another strange issue in v0.8.0: If there is already video proceeded in the folder, it seems that re-process another video in the same folder will result in strange coordinate results in mapillary_image_description.json.

I have some sequences that is uploaded with completely wrong coordinates :(

[ptpt]

@littlebtc which command did you rerun? If the video path is a single video file (e.g. ~/todo/FILE210916-094020F.MP4), only the sampled images from this file will be processed.

By strange coordiantes, do you mean it geotag wrong videos from the nmea file, or the coordinates interpolated from the nmea file are strange?

I usually run the following command to debug the image description file:

python3 -m mapillary_tools.geojson YOUR_VIDEO_PATH/mapillary_sampled_video_frames/mapillary_image_description.json | pbcopy
# paste the output in geojson.io

Would be great if you could share some sample files for debugging.

Thanks for trying out the new tools.

[ptpt]

The incorrect locations are likely to relate to https://github.com/mapillary/mapillary_tools/issues/441 and it's fixed.

Could you run this, see is there any creation time there:

ffprobe -show_streams YOUR_VIDEO

[ptpt]

@Bartixxx32 The video creation time is required for sample image location interpolation. The reason 0.7 works is that it uses 0 as the default timestamp if creation time is not found.

In 0.8 you can do the same manually:

python3 -m mapillary_tools.commands video_process --video_start_time 1970_01_01_00_00_00_000 YOUR_VIDEO.mp4 --geotag_source "nmea" --geotag_source_path todo/FILE210225-222951F.NMEA

However, you might see lots of images failed due to OutOfGPX error. This is because 1970_01_01_00_00_00_000 is out of your GPX time range. In this case, use --interpolation_use_gpx_start_time to interpolate your first sample image on the start point of your GPX, and see how it goes.

[Bartixxx32]

ok seems, this one worked thanks a lot, btw i have other question.

I`m able to process multiple files at once? like 10 videos in vid/ folder and 10 nmea in nmea/ folder, will program know whih video and nmea it should merge? nmea and vid have same names

[ptpt]

I`m able to process multiple files at once? like 10 videos in vid/ folder and 10 nmea in nmea/ folder, will program know whih video and nmea it should merge? nmea and vid have same names

No unfortunately, mapillary_tools isn't very smart. In this case you can script it, for example:

for video in vid/*.mp4; do
    base=$(basename $video .mp4)
    mapillary_tools video_process vid/${base}.mp4 --geotag_source_path nmea/${base}.nmea
done

[Bartixxx32]

oh, thats gonna work :) but in wsl mode ill will get slower convertions but, its better than nothing! Thanks

[Bartixxx32]

Hello @ptpt , i have ended with somentig like: But doing mapillary_tools upload vid/mapillary_sampled_video_frames/ seems only uploading the one sequence :/

My command was:

for video in vid/*.MP4; do
    base=$(basename $video .MP4)
    mapillary_tools video_process --video_start_time 1970_01_01_00_00_00_000 vid/${base}.mp4 --geotag_source "nmea" --geotag_source_path nmea/${base}.nmea --interpolation_use_gpx_start_time
done

I guess it will not work when all is in same directory i guess the json file is recreated every new video is processed :/

I think i have workaround, but not sure for now, its sad i lost those processed data :P

Updated command:

for video in vid/*.MP4; do
    base=$(basename $video .MP4)
    mapillary_tools video_process --video_start_time 1970_01_01_00_00_00_000 vid/${base}.mp4 obrazki/${base} --geotag_source "nmea" --geotag_source_path nmea/${base}.nmea --interpolation_use_gpx_start_time
done

Structure:

[ptpt]

I guess it will not work when all is in same directory i guess the json file is recreated every new video is processed :/

That's right. video_process is simply the combo of sample_video and process. process will generate the new mapillary description file and override the old one if exists. Sorry for the lost processed data.

When you script your upload solution, it might be useful to test the upload part with --dry_run first, which simulates the upload and tell you how many images/videos will be uploaded.

The workaround is legit 👍 .