sleap icon indicating copy to clipboard operation
sleap copied to clipboard

AV1 support

Open talmo opened this issue 2 years ago • 0 comments

AV1 is a new modern video compression codec that may be an appealing option for use in acquisition systems, so we should prioritize supporting it as both we and others will consider using it for future datasets.

We currently use OpenCV with the ffmpeg backend for video decoding. In addition to some issues with seeking reliability, it seems that the version of ffmpeg linked into the version of OpenCV we're using doesn't seem to support AV1.

Here's a couple of commands to encode a test video into AV1 and H264:

ffmpeg -ss 00:00 -i test.avi -t 00:20 -vf hqdn3d=4:4:3:3 -c:v libsvtav1 -crf 35 test_av1_CRF35_denoised.mp4
ffmpeg -ss 00:00 -i test.avi -t 00:20 -vf hqdn3d=4:4:3:3 -c:v libx264 -crf 23 test_h264_CRF23_denoised.mp4

When loading videos encoded with AV1, we get the following errors:

[av1 @ 0000021206964c80] Missing Sequence Header.
[av1 @ 0000021261b67700] Your platform doesn't suppport hardware accelerated AV1 decoding.
[av1 @ 0000021261b67700] Failed to get pixel format.
Unable to load video with these parameters. Error: "Unable to load frame 0 from MediaVideo(filename='C:/Users/Talmo/Downloads/test_av1_CRF35_denoised.mp4', grayscale=False, bgr=True, dataset='', input_format='')."

This is on Windows 10 with:

# Name                    Version                   Build  Channel
ffmpeg                    4.2.2                he774522_0         # shouldn't matter since we use the built-in one in opencv?
opencv-python             4.5.5.64                 pypi_0    pypi

Checking the version we have compiled with opencv:

python -c "import cv2; print(cv2.getBuildInformation())"

Output:

...
  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (prebuilt binaries)
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES
...

On an M1 Mac, however, we are able to decode AV1 files! Here are the versions:

# Name                    Version                   Build  Channel
ffmpeg                    4.4.2           gpl_hf318d42_112    conda-forge    # Again, shouldn't matter, but just in case
libopencv                 4.6.0            py39hb16ced2_2    conda-forge
opencv                    4.6.0            py39hdf13c20_2    conda-forge
py-opencv                 4.6.0            py39hfa6204d_2    conda-forge

And build info:

  Video I/O:
    FFMPEG:                      YES
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)
    AVFoundation:                YES

The steps to fix this would be to either update opencv-python to >= 4.6.0, or to switch to a different backend for video I/O entirely (like pyav or imageio).

Issue report credit: @olinesn

talmo avatar Jun 20 '23 01:06 talmo