Correction of FrameUtility

[moi15moi]

Finally, this should be the last version of FrameUtility. For reference, here is the previous version: #37

Why it is needed?

The previous FrameUtility would not work with VFR video, now yes.

Finally, the previous FrameUtility was more a hack than anything else. Now it relies on the same algorithm as Aegisub.

What has been done?

• Add TimeType Enum
• Add decord dependency
• Be able to create Timestamps with 3 methods (from_fps, from_video_file, from_timestamps_file)
• Correction how to convert ms to ass_timestamps
• Correction of the algorithm of ms_to_frames and frames_to_ms
• Add some tests

How to get the timestamps from an video?

I did some test with multiple library.

• decord
• MoviePy
• PyAV
• pyffms2
• OpenCV

I choose to use decord, because it was the fastest one and it is easy to install compared to some of them like pyffms2.

Important to note, decord seems to automatically normalize the timestamps.

import av
import cv2
import time
from decord import VideoReader
from ffms2 import VideoSource
from moviepy.editor import VideoFileClip
from typing import List


def with_movie_py(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/moviepy/
    My comments:
        The timestamps I get are not good compared to gMKVExtractGUI or ffms2. (I only tried with VFR video)

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    vid = VideoFileClip(video)

    timestamps = [
        round(tstamp * 1000) for tstamp, frame in vid.iter_frames(with_times=True)
    ]

    return timestamps


def with_cv2(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/opencv-python/
    My comments:
        I don't know why, but the last 4 or 5 timestamps are equal to 0 when they should not.
        Also, cv2 is slow. It took my computer 132 seconds to process the video.


    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    timestamps = []
    cap = cv2.VideoCapture(video)

    while cap.isOpened():
        frame_exists, curr_frame = cap.read()
        if frame_exists:
            timestamps.append(round(cap.get(cv2.CAP_PROP_POS_MSEC)))
        else:
            break

    cap.release()

    return timestamps


def with_pyffms2(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/ffms2/
    My comments:
        Works really well, but it doesn't install ffms2 automatically, so you need to do it by yourself.
        The easiest way is to install Vapoursynth and use it to install ffms2.
        Also, the library doesn't seems to be really maintained.

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    video_source = VideoSource(video)

    # You can also do: video_source.track.timecodes
    timestamps = [
        int(
            (frame.PTS * video_source.track.time_base.numerator)
            / video_source.track.time_base.denominator
        )
        for frame in video_source.track.frame_info_list
    ]

    return timestamps


def with_decord(video: str) -> List[int]:
    """
    Link: https://github.com/dmlc/decord
    My comments:
        Works really well, but it seems to only work with mkv and mp4 file.
        Mp4 file can have a +- 1 ms difference with ffms2, but it is acceptable.

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    vr = VideoReader(video)

    timestamps = vr.get_frame_timestamp(range(len(vr)))
    timestamps = (timestamps[:, 0] * 1000).round().astype(int).tolist()

    return timestamps


def with_pyav(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/av/
    My comments:
        I don't know why, but sometimes it simply doesn't work with certain video.
        Also, I tested with different mp4 file and sometimes it take 8 seconds and sometimes 117 seconds.

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    container = av.open(video)
    video = container.streams.video[0]
    av_timestamps = [
        int(frame.pts * video.time_base * 1000) for frame in container.decode(video)
    ]

    container.close()

    return av_timestamps


def main():
    video = r"WRITE_YOUR_VIDEO_PATH"

    start = time.process_time()
    movie_py_timestamps = with_movie_py(video)
    print(f"With Movie py {time.process_time() - start} seconds")

    start = time.process_time()
    cv2_timestamps = with_cv2(video)
    print(f"With cv2 {time.process_time() - start} seconds")

    start = time.process_time()
    ffms2_timestamps = with_pyffms2(video)
    print(f"With ffms2 {time.process_time() - start} seconds")

    start = time.process_time()
    decord_timestamps = with_decord(video)
    print(f"With decord {time.process_time() - start} seconds")

    start = time.process_time()
    av_timestamps = with_pyav(video)
    print(f"With av {time.process_time() - start} seconds")


if __name__ == "__main__":
    main()

Here is how much times it took to get the timestamps for an mkv of 24 minutes.

With Movie py 11.71875 seconds
With cv2 23.359375 seconds
With ffms2 0.3125 seconds
With decord 0.125 seconds
With av 8.75 seconds