Bug in FFmpegReader class leading to faulty HW Accel initialization when CUDA is chosen

[teito-dev]

The FFmpegReader.cpp file has a bug in get_hw_dec_format

I've been trying to render a video with nvidia cuda in libopenshot with HARDWARE_DECODER = 2 and the following setup

openshot::Settings::Instance()->HARDWARE_DECODER = 2;
openshot::Settings::Instance()->HW_DE_DEVICE_SET = 0;
openshot::Settings::Instance()->HW_EN_DEVICE_SET = 0;
    
openshot::Settings::Instance()->OMP_THREADS = 12;
openshot::Settings::Instance()->DE_LIMIT_HEIGHT_MAX = 1100;
openshot::Settings::Instance()->DE_LIMIT_WIDTH_MAX = 1950;

// Then im using the following options when exporting the video
if (useGpu) {
            writer.SetVideoOptions(true, "h264_nvenc", fps, width, height, openshot::Fraction(1, 1), false, false, 2000000);
            writer.SetAudioOptions(true, "aac", sample_rate, channels, channel_layout, 128000);
            writer.PrepareStreams();
            writer.SetOption(openshot::VIDEO_STREAM, "preset", "p1");
            writer.SetOption(openshot::VIDEO_STREAM, "tune", "ll");
 }

However ive been getting this error that crashes it :

The issue is in the get_hw_dec_format() function at FFmpegReader.cpp:127-185. This function is a callback that FFmpeg calls to choose the pixel format, and it's overriding my HARDWARE_DECODER setting.

The Problem:

1. I had set HARDWARE_DECODER = 2 (CUDA) in my switch statement at line 312
2. This correctly sets hw_de_av_device_type = AV_HWDEVICE_TYPE_CUDA
3. BUT the get_hw_dec_format() callback function checks what pixel formats are available from the decoder
4. On Linux, it checks in this order: AV_PIX_FMT_VAAPI (line 135), then AV_PIX_FMT_VDPAU (line 140), then AV_PIX_FMT_CUDA (line 168)
5. If the H.264 decoder supports VDPAU, it returns AV_PIX_FMT_VDPAU and overwrites the global variables: hw_de_av_pix_fmt_global = AV_PIX_FMT_VDPAU; // ← This overwrites your CUDA setting! hw_de_av_device_type_global = AV_HWDEVICE_TYPE_VDPAU; // ← This too!
6. Later at line 1186, these global variables are used instead of your setting

The Fix: We need to modify the get_hw_dec_format() function to respect the HARDWARE_DECODER setting instead of just taking the first available format. The function should check the preference first before falling back to what's available.

I've fixed it on my version and it does seem to work overall , however im not sending a pull request yet as i havent run all the unit tests on the new updated code for FFmpegReader.cpp , however it certainly has solved my initial problem

Let me know if im wrong, im new to the codebase and so I'm a bit afraid to send a pull request yet, without getting feedback from one of the libopenshot devs here.

Also thanks a lot for making this project !

[Colorjet3]

Hello @jonoomph please review this.

[aperture147]

Hi is this issue any related to #956? I can't build a standalone libopenshot with working HW accel.