AV1 support
AV1 is a new modern video compression codec that may be an appealing option for use in acquisition systems, so we should prioritize supporting it as both we and others will consider using it for future datasets.
We currently use OpenCV with the ffmpeg backend for video decoding. In addition to some issues with seeking reliability, it seems that the version of ffmpeg linked into the version of OpenCV we're using doesn't seem to support AV1.
Here's a couple of commands to encode a test video into AV1 and H264:
ffmpeg -ss 00:00 -i test.avi -t 00:20 -vf hqdn3d=4:4:3:3 -c:v libsvtav1 -crf 35 test_av1_CRF35_denoised.mp4
ffmpeg -ss 00:00 -i test.avi -t 00:20 -vf hqdn3d=4:4:3:3 -c:v libx264 -crf 23 test_h264_CRF23_denoised.mp4
When loading videos encoded with AV1, we get the following errors:
[av1 @ 0000021206964c80] Missing Sequence Header.
[av1 @ 0000021261b67700] Your platform doesn't suppport hardware accelerated AV1 decoding.
[av1 @ 0000021261b67700] Failed to get pixel format.
Unable to load video with these parameters. Error: "Unable to load frame 0 from MediaVideo(filename='C:/Users/Talmo/Downloads/test_av1_CRF35_denoised.mp4', grayscale=False, bgr=True, dataset='', input_format='')."
This is on Windows 10 with:
# Name Version Build Channel
ffmpeg 4.2.2 he774522_0 # shouldn't matter since we use the built-in one in opencv?
opencv-python 4.5.5.64 pypi_0 pypi
Checking the version we have compiled with opencv:
python -c "import cv2; print(cv2.getBuildInformation())"
Output:
...
Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.134.100)
avformat: YES (58.76.100)
avutil: YES (56.70.100)
swscale: YES (5.9.100)
avresample: YES (4.0.0)
GStreamer: NO
DirectShow: YES
Media Foundation: YES
DXVA: YES
...
On an M1 Mac, however, we are able to decode AV1 files! Here are the versions:
# Name Version Build Channel
ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge # Again, shouldn't matter, but just in case
libopencv 4.6.0 py39hb16ced2_2 conda-forge
opencv 4.6.0 py39hdf13c20_2 conda-forge
py-opencv 4.6.0 py39hfa6204d_2 conda-forge
And build info:
Video I/O:
FFMPEG: YES
avcodec: YES (58.134.100)
avformat: YES (58.76.100)
avutil: YES (56.70.100)
swscale: YES (5.9.100)
avresample: YES (4.0.0)
AVFoundation: YES
The steps to fix this would be to either update opencv-python to >= 4.6.0, or to switch to a different backend for video I/O entirely (like pyav or imageio).
Issue report credit: @olinesn