SIGSEGV upon decoding some images with libde265

[mnaydenov]

Hello, I am experiencing Segmentation fault when decoding these images: heic-segfault.zip

Steps to reproduce

Run the suggested in the Readme.md example code, to read an image, with any of the files from the zip.

Environment

libde265 @ 900772c3e9ee1e106b93283fd8e7633d52899e40 libheif @ 9453b49ea8e9b7a0ad022650ce4b386864d2dac3 Edit: Also tested latest 5f948947733bceb5ff243c785265ebeeabbeb170

Both compiled with -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED_LIBS=NO.

Edition Windows 10 Pro Version 21H1 Installed on ‎13.‎11.‎2020 ‎г. OS build 19043.1165 Experience Windows Feature Experience Pack 120.2212.3530.0

Compiler is mingw-w64/i686-5.3.0-posix-dwarf-rt_v4-rev0

Debugger console

[New Thread 28408.0x1a4c]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 28408.0x1a4c]
ff_hevc_transform_16x16_add_8_sse4 (_dst=0x348e0d0 '�' <repeats 16 times>, coeffs=0x367b020, _stride=1280) at libde265/x86/sse-dct.cc:1524
1524	    const __m128i T00 = _mm_load_si128((__m128i *) (transform16x16_1[0][0]));
=thread-selected,id="5"

Stacktrace

ff_hevc_transform_16x16_add_8_sse4(uint8_t * _dst, const int16_t * coeffs, ptrdiff_t _stride) (libde265\x86\sse-dct.cc:1524)
acceleration_functions::transform_add<unsigned char>(const acceleration_functions * const this, int sizeIdx, uint8_t * dst, const int16_t * coeffs, ptrdiff_t stride, int bit_depth) (libde265\acceleration.h:353)
transform_coefficients<unsigned char>(acceleration_functions * acceleration, int16_t * coeff, int coeffStride, int nT, int trType, unsigned char * dst, int dstStride, int bit_depth) (libde265\transform.cc:230)
scale_coefficients_internal<unsigned char>(thread_context * tctx, int xT, int yT, int x0, int y0, int nT, int cIdx, bool transform_skip_flag, bool intra, int rdpcmMode) (libde265\transform.cc:609)
scale_coefficients(thread_context * tctx, int xT, int yT, int x0, int y0, int nT, int cIdx, bool transform_skip_flag, bool intra, int rdpcmMode) (libde265\transform.cc:649)
decode_TU(thread_context * tctx, int x0, int y0, int xCUBase, int yCUBase, int nT, int cIdx, PredMode cuPredMode, bool cbf) (libde265\slice.cc:3472)
read_transform_unit(thread_context * tctx, int x0, int y0, int xBase, int yBase, int xCUBase, int yCUBase, int log2TrafoSize, int trafoDepth, int blkIdx, int cbf_luma, int cbf_cb, int cbf_cr) (libde265\slice.cc:3665)
read_transform_tree(thread_context * tctx, int x0, int y0, int xBase, int yBase, int xCUBase, int yCUBase, int log2TrafoSize, int trafoDepth, int blkIdx, int MaxTrafoDepth, int IntraSplitFlag, PredMode cuPredMode, uint8_t parent_cbf_cb, uint8_t parent_cbf_cr) (libde265\slice.cc:3964)
read_coding_unit(thread_context * tctx, int x0, int y0, int log2CbSize, int ctDepth) (libde265\slice.cc:4577)
read_coding_quadtree(thread_context * tctx, int x0, int y0, int log2CbSize, int ctDepth) (libde265\slice.cc:4652)
read_coding_quadtree(thread_context * tctx, int x0, int y0, int log2CbSize, int ctDepth) (libde265\slice.cc:4635)
read_coding_quadtree(thread_context * tctx, int x0, int y0, int log2CbSize, int ctDepth) (libde265\slice.cc:4638)
read_coding_tree_unit(thread_context * tctx) (libde265\slice.cc:2861)
decode_substream(thread_context * tctx, bool block_wpp, bool first_independent_substream) (libde265\slice.cc:4741)
thread_task_ctb_row::work(thread_task_ctb_row * const this) (libde265\slice.cc:4991)
worker_thread(LPVOID pool_ptr) (libde265\threads.cc:233)
kernel32.dll!KERNEL32!BaseThreadInitThunk (Unknown Source:0)
ntdll.dll!ntdll!RtlGetAppContainerNamedObjectPath (Unknown Source:0)
ntdll.dll!ntdll!RtlGetAppContainerNamedObjectPath (Unknown Source:0)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)

---

Issue is NOT present under Linux The exact codebase is compiled under Ubuntu 20.04, using the exact same options (as far as cmake is concerned), and the issue was not reproduced.

---

Issue is NOT present when compiling with MSVC (both x64 and x86)

Looks like the problem is specific to the combination MinGW + Windows.

Consequently, there is a workaround if one compiles libde265 as a shared library with MSVC, then compile the rest with MinGW and link to it. Tested and it works.

---

Issue is NOT present when libde265 is compiled with DISABLE_SSE This should be obvious, considering the code is clearly in the optimized image processing. Still, this is tested and confirmed.