Investigate `ffmpegcv` backend for video input

[Breakthrough]

See if https://github.com/chenxinfeng4/ffmpegcv is worth adding as a backend. It might be useful in the future, especially as it supports GPU (CUDA).

[Breakthrough]

It seems this library might not do end-to-end GPU decoding unfortunately, but we might want to consider handling an ffmpeg decode pipe ourselves.

On Sun, Apr 27, 2025 at 5:47 PM Nilas @.***> wrote:

NevermindNilas left a comment (Breakthrough/PySceneDetect#505) https://github.com/Breakthrough/PySceneDetect/issues/505#issuecomment-2833654714

In theory it's just a fancy wrapper around ffmpeg's -f rawvideo piping, you can achieve similar results by calling a subprocess.popen and piping the stdout to a variable. Cuda accel is achieved through -hwaccel cuda.

— Reply to this email directly, view it on GitHub https://github.com/Breakthrough/PySceneDetect/issues/505#issuecomment-2833654714, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA6TBFBLWZLTHGCMQIZPSL23VF63AVCNFSM6AAAAAB3EM32FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZTGY2TINZRGQ . You are receiving this because you authored the thread.Message ID: @.***>

[NevermindNilas]

Hi @Breakthrough

Moviepy afaik already integrates a similar approach to piping ffmpeg's rawvideo stdout.

But if you wish to add it yourself you have to make sure that all color formats are supported which is a pain, assuming your internal pipeline works with rgb24 / bgr24, in that case yuv420p, rgb24 or bgr24 will work just fine because FFMPEG can internally change the color format to the desired color format, but the moment you have yuv420p10le or yuv444p inputs the pipeline breaks. Maybe you could use kornia's built in functions in order to accurately estimate the bytesizes of each frame but as you can imagine, it's a pretty error prone / fragile workflow.

I have a similar approach right here altho' deprecated for a good while now due to the reasons mentioned above:

https://github.com/NevermindNilas/TheAnimeScripter/blob/ced1cf3ac6da5a96b40d1c917168a3e47a844a45/src%2FffmpegSettings.py#L108

If you wish to add such workflow lmk as I am willing to contribute to the best of my knowledge, but I'd suggest sticking with pyav for the simplicity and look for other opportunities to improve performance.