[macOS] [30.1 Beta] Starting AOM AV1 output crashes OBS

[jcm93]

Operating System Info

macOS 14

OBS Studio Version

30.1.0-beta1

OBS Studio Log URL

https://obsproject.com/logs/794vCPjjyYfysnRo

OBS Studio Crash Log URL

https://github.com/obsproject/obs-studio/files/14225366/crash.txt

Expected Behavior

OBS not to crash when doing AV1 output.

Current Behavior

OBS crashes when starting any kind of AV1 output

Steps to Reproduce

1. Install the 30.1 beta
2. Attempt to start output with the software AOM AV1 encoder

Anything else we should know?

I did a bit of sleuthing on this issue yesterday.

[!IMPORTANT] Re-signing the release build ad-hoc with codesign --force --deep -s - OBS.app makes the issue go away, which is quite puzzling. I suspect this issue has something to do with the way the libavcodec binary is signed by OBS.

I tried to reproduce the crash in a self-compiled OBS builds or in debug mode within the IDE in an attempt to get a proper backtrace, but the issue did not occur there either, which made me suspicious. So I manually symbolicated the crashed thread, which revealed the following:

Thread 19 Crashed:: aom enc worker
0   libavcodec.dylib                         av1_convolve_x_sr_neon_i8mm (in libavcodec.dylib) (convolve_neon_i8mm.c:298)
1   libavcodec.dylib                         av1_make_inter_predictor (in libavcodec.dylib) (reconinter.c:121)
2   libavcodec.dylib                         av1_enc_build_inter_predictor (in libavcodec.dylib) (reconinter_enc.c:118)
3   libavcodec.dylib                         encode_superblock (in libavcodec.dylib) (partition_search.c:464)
4   libavcodec.dylib                         encode_b_nonrd (in libavcodec.dylib) (partition_search.c:2104)
5   libavcodec.dylib                         av1_nonrd_use_partition (in libavcodec.dylib) (partition_search.c:0)
6   libavcodec.dylib                         av1_nonrd_use_partition (in libavcodec.dylib) (partition_search.c:0)
7   libavcodec.dylib                         av1_encode_sb_row (in libavcodec.dylib) (encodeframe.c:1406)
8   libavcodec.dylib                         enc_row_mt_worker_hook (in libavcodec.dylib) (ethread.c:0)
9   libavcodec.dylib                         thread_loop (in libavcodec.dylib) (aom_thread.c:78)
10  libsystem_pthread.dylib                  0x199602fa0 _pthread_start + 136
11  libsystem_pthread.dylib                  0x1995fdd40 thread_start + 8

This was also somewhat puzzling/unrevealing, at least to me.

Edit: Also importantly, other parts of libavcodec seem to be working fine. I can use the Custom FFmpeg Output mode with x264 or hardware encoders without issues, seemingly.

Full (unsymbolicated) crash log here: crash.txt

[PatTheMav]

Crash actually occurs in static INLINE int16x8_t convolve8_8_2d_h(uint8x16_t samples, const int8x8_t filters, const uint8x16x3_t permute_tbl, const int32x4_t horiz_const) on this instruction:

sum[0] = vusdotq_lane_s32(horiz_const, permuted_samples[0], filters, 0);

Seems like filters is a NULL pointer and that obviously crashes the vector instruction.

[RytoEX]

Technically not fixed until we actually update the deps here.