FFmpeg videoio backend won't decode low fps videos on machines with lots of cpus

[jcowgill]

System information (version)

• OpenCV => 4.5.2
• Operating System / Platform => Ubuntu 18.04
• Compiler => GCC 7.5

Detailed description

The attached test videos can't be decoded by VideoCapture using the FFmpeg backend in OpenCV (grab always returns false). They both play correctly with ffplay.

I generated both the videos by downloading Spring (https://www.youtube.com/watch?v=WhWc3b3KhnY - probably another video would work), then running these ffmpeg commands to reduce the framerate:

ffmpeg -ss 60 -t 100 -i Spring.mkv -codec:a copy -filter:v fps=10 -crf 30 test-10fps.mkv
ffmpeg -ss 60 -t 100 -i Spring.mkv -codec:a copy -filter:v fps=0.1 test-01fps.mkv

(-crf 30 is only there to not hit the upload size limit)

I have two test machines: a laptop and a big server.

• On the first machine which has 8 logical cpus, the test-10fps.mkv video works, but the test-01fps.mkv video does not.
• On the second machine which has 128 logical cpus both videos fail. If I offline 96 of the cpus (32 remaining, no code or any other changes) the test-10fps.mkv video starts working.

I also noticed that:

• Removing the audio streams from either file makes them work correctly.
• Sortening the file too much also makes them work.

Steps to reproduce

#include <opencv2/opencv.hpp>
#include <iostream>

int main(int argc, char**argv)
{
    auto x = cv::VideoCapture(argv[1], cv::CAP_FFMPEG);
    std::cout << (x.grab() ? "true" : "false") << std::endl;
    return 0;
}

$ g++ test.cpp -I /usr/local/include/opencv4/ -L /usr/local/lib/ -lopencv_videoio -lopencv_core
$ ./a.out test-1fps.mkv
false

Should print true.

Test files

https://user-images.githubusercontent.com/1226825/116454905-5dd8a680-a858-11eb-8323-c5729ba03016.mp4

https://user-images.githubusercontent.com/1226825/116454945-6a5cff00-a858-11eb-945f-273687c9c00b.mp4

Issue submission checklist

• [x] I report the issue, it's not a question
• [x] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found solution
• [x] I updated to latest OpenCV version and the issue is still there
• [x] There is reproducer code and related data files: videos, images, onnx, etc (However I expect I could find a smaller reproducer if I tried very hard)

[asmorkalov]

@jcowgill Thanks for the issue report. Could you please provide you CMakeCache.txt file and information about ffmpeg version. Also could you try to play the video with mplayer with corresponding version. It uses FFMpeg for decoding.

[jcowgill]

I used this to compile OpenCV:

$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DWITH_IPP=N -DWITH_ADE=N -DBUILD_LIST=core,videoio -G Ninja
$ ninja
$ ninja install

CMakeCache.txt

$ ffmpeg -version
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100

On my local machine I've tried with ffplay, mplayer, mpv and totem. Both files work for me in all those apps. I also notice both files play in Firefox when I view this Github issue (but I have no idea what backend Firefox is using).

[jcowgill]

I've been doing some experiments and I've noticed that:

This change is enough to make both videos fail to play on my laptop (previously only the 0.1 fps one would fail):

diff --git a/modules/videoio/src/cap_ffmpeg_impl.hpp b/modules/videoio/src/cap_ffmpeg_impl.hpp
index 84e4e722f7..ed0716d3da 100644
--- a/modules/videoio/src/cap_ffmpeg_impl.hpp
+++ b/modules/videoio/src/cap_ffmpeg_impl.hpp
@@ -307,7 +307,7 @@ static int get_number_of_cpus(void)
 
     return (int)sysinfo.dwNumberOfProcessors;
 #elif defined __linux__ || defined __HAIKU__
-    return (int)sysconf( _SC_NPROCESSORS_ONLN );
+    return 128;
 #elif defined __APPLE__
     int numCPU=0;
     int mib[4];

This seems to fix both videos for me:

diff --git a/modules/videoio/src/cap_ffmpeg_impl.hpp b/modules/videoio/src/cap_ffmpeg_impl.hpp
index 84e4e722f7..65095d9333 100644
--- a/modules/videoio/src/cap_ffmpeg_impl.hpp
+++ b/modules/videoio/src/cap_ffmpeg_impl.hpp
@@ -1317,9 +1317,6 @@ bool CvCapture_FFMPEG::grabFrame()
         if( packet.stream_index != video_stream )
         {
             _opencv_ffmpeg_av_packet_unref (&packet);
-            count_errs++;
-            if (count_errs > max_number_of_attempts)
-                break;
             continue;
         }
 

I think what is happening is a combination of:

• FFmpeg buffering a lot of packets when the number of threads is set to a high value.
• My test videos being a low framerate which increases the ratio of audio packets to video packets.

Both these make it more likely to hit the max_number_of_attempts limit and cause the function to fail.

While the above change works for me, I really have no idea if it's the right fix because I don't know why the attempts limit exists in the first place.

[Mazuzel]

Hi ! Setting ffmpeg thread count with the number of CPUs doesn't seem relevant when we want to decode a video on a server (sharing a lot of CPUs with other processes). Should we add a way to limit or set the number of decoding threads ?

[luxifeo]

Hi, I have the same issue too.

In this line, OpenCV sets the number of threads equal to the number of logical CPUs

https://github.com/opencv/opencv/blob/139c44377032f58849bbab2ae454fcf14e89d762/modules/videoio/src/cap_ffmpeg_impl.hpp#L985-L992

There is an argument threads which can be parsed from OPENCV_FFMPEG_CAPTURE_OPTIONS. Can we get this value to set the number of threads FFMPEG uses?

[cudawarped]

@luxifeo Can you see if https://github.com/opencv/opencv/pull/22248 fixes your issue when reading from an RTSP stream and if so can you check if you have the same issue when reading from a file on a machine with a large number of CPU cores and if thid fixes that as well?

[luxifeo]

@luxifeo Can you see if #22248 fixes your issue when reading from an RTSP stream and if so can you check if you have the same issue when reading from a file on a machine with a large number of CPU cores and if thid fixes that as well?

Sorry for the late reply. I confirm your solution worked with RTSP Stream. Current OpenCV version still works fine with low FPS video, however.

[asmorkalov]

@luxifeo I assume that the issue could be closed. right?

[luxifeo]

@luxifeo I assume that the issue could be closed. right?

I tried cudawarped's PR and it worked. Btw, I am not the issue's author. Should you ask them too?

[asmorkalov]

@luxifeo Thanks a lot for the conformation!