Segfault when recording with GPU texture encoder with color format other that NV12 or P010

[tomben13]

Operating System Info

Other

Other OS

ArchLinux

OBS Studio Version

32.0.0

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/zQofTlOJNhL7bbzH

OBS Studio Crash Log URL

No response

Expected Behavior

When selecting a color format other than NV12 or P010 with a GPU that's capable of texture encoding, I expect to have an error message that does not allow me to select it.

Current Behavior

Crash with segmentation fault when starting the recording.

The crash log with backtrace is below. I have analyzed the issue with GDB and it comes down to this :

• gpu_encode_available() returns false in add_connection() because of the color format setting.
• therefore we call start_raw_video() with receive_video() as the callback
• receive_video() calls do_encode() which expects the obs_encoder_info struct to have an encode() method.
• texture encoders do not have this method (NULL), therefore it segfault at this point

If I use NV12 color format, there is no issue.

info: ---------------------------------
info: [FFmpeg VAAPI encoder: 'advanced_video_recording'] settings:
	device:       /dev/dri/by-path/pci-0000:03:00.0-render
	rate_control: CQP
	profile:      0
	level:        -99
	qp:           20
	bitrate:      0
	maxrate:      0
	keyint:       120
	width:        1920
	height:       1080
	b-frames:     0
	ffmpeg opts:

[New Thread 0x7fff89ffb6c0 (LWP 7039)]
[New Thread 0x7fff75a776c0 (LWP 7040)]
[New Thread 0x7fff8a9ff6c0 (LWP 7041)]
[New Thread 0x7fff767bd6c0 (LWP 7042)]
[New Thread 0x7fff88a756c0 (LWP 7043)]
[New Thread 0x7fff7727a6c0 (LWP 7044)]
[New Thread 0x7fff752766c0 (LWP 7045)]
info: libfdk_aac encoder created
info: libfdk_aac bitrate: 160, channels: 2
info: libfdk_aac encoder created
info: libfdk_aac bitrate: 160, channels: 2
info: libfdk_aac encoder created
info: libfdk_aac bitrate: 160, channels: 2
info: libfdk_aac encoder created
info: libfdk_aac bitrate: 160, channels: 2
[Detaching after vfork from child process 7046]
info: ==== Recording Start ===============================================
info: [ffmpeg muxer: 'adv_file_output'] Writing file '/home/tom/Videos/obs/2025-10-14_20-33-58.mkv'...
[New Thread 0x7fff6fd076c0 (LWP 7047)]
[New Thread 0x7fff6f5066c0 (LWP 7048)]

Thread 21 "video-io: video" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa3fff6c0 (LWP 6745)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff5129a8c in do_encode (encoder=encoder@entry=0x5555575b1800, frame=frame@entry=0x7fffa3ffe020,
    frame_cts=frame_cts@entry=0x7fffa3ffe160) at /usr/src/debug/obs-studio/obs-studio-32.0.1-sources/libobs/obs-encoder.c:1417
#2  0x00007ffff5129e61 in receive_video (param=0x5555575b1800, frame=0x7fffa3ffe100)
    at /usr/src/debug/obs-studio/obs-studio-32.0.1-sources/libobs/obs-encoder.c:1522
#3  0x00007ffff51a23f7 in video_output_cur_frame (video=0x555556907aa0)
    at /usr/src/debug/obs-studio/obs-studio-32.0.1-sources/libobs/media-io/video-io.c:159
#4  video_thread (param=0x555556907aa0) at /usr/src/debug/obs-studio/obs-studio-32.0.1-sources/libobs/media-io/video-io.c:204
#5  0x00007ffff24969cb in start_thread (arg=<optimized out>) at pthread_create.c:448
#6  0x00007ffff251aa0c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb)

Steps to Reproduce

...

Anything else we should know?

No response

[v0l]

At least do_encode should check for nullptr encode fn