Extreme memory usage during JXL decode (GraphicsMagick oss-fuzz 69728 test case #2)

[bobfriesenhahn]

Describe the bug

Libjxl consumes huge amounts of heap memory ( 3,393,388,337 bytes) while decoding this tiny (127 bytes!) file

With a somewhat older libxl, there was also this unusual exception:

zsh: illegal hardware instruction (core dumped)

To Reproduce

djxl 5791454446813184-tc2.jxl crap.pnm

With this test case:

Expected behavior

Libjxl should detect an irrational request and reject decoding prior to allocating huge amounts of memory.

Screenshots

% LD_PRELOAD=/lib/x86_64-linux-gnu/libmemusage.so djxl 5791454446813184-tc2.jxl crap.pnm
JPEG XL decoder v0.11.0 ba759b42 [AVX2,SSE4,SSE2]
Failed to decode image
DecompressJxlToPackedPixelFile failed

Memory usage summary: heap total: 997603716677, heap peak: 3393388337, stack peak: 7552
         total calls   total memory   failed calls
 malloc|        639   997603709957              1
realloc|          0              0              0  (nomove:0, dec:0, free:0)
 calloc|         12           6720              0
   free|        729     3393410181
Histogram for block sizes:
    0-15            105  16% ==================================================
   16-31             92  14% ===========================================
   32-47             60   9% ============================
   48-63             32   4% ===============
   64-79             63   9% ==============================
   80-95             28   4% =============
   96-111             4  <1% =
  112-127             1  <1% 
  128-143            72  11% ==================================
  160-175             3  <1% =
  176-191             2  <1% 
  192-207             4  <1% =
  240-255             2  <1% 
  256-271            42   6% ====================
  304-319             2  <1% 
  320-335             9   1% ====
  368-383             1  <1% 
  464-479             1  <1% 
  512-527            20   3% =========
  528-543            39   5% ==================
  576-591             2  <1% 
  720-735            16   2% =======
  752-767             1  <1% 
  880-895             1  <1% 
  992-1007            1  <1% 
 1024-1039            2  <1% 
 1040-1055            4  <1% =
 1072-1087           13   1% ======
 1440-1455            2  <1% 
 1536-1551            1  <1% 
 1648-1663            1  <1% 
 2176-2191            2  <1% 
 2304-2319            7   1% ===
 2816-2831            1  <1% 
 2960-2975            1  <1% 
 3200-3215            1  <1% 
 4096-4111            1  <1% 
 4352-4367            2  <1% 
 4560-4575            1  <1% 
 5504-5519            1  <1% 
 7808-7823            3  <1% =
 9520-9535            2  <1% 
   large              3  <1% =

Environment

• OS: Ubuntu 22.04 LTS
• Compiler version: clang 14.0.0-1
• CPU type: x86_64
• cjxl/djxl version string: cjxl v0.11.0 ba759b42 [AVX2,SSE4,SSE2]

Additional context

None

[Traneptora]

Image Size: 848302208x293

To hold the uncompressed image in memory requires 745,657,640,832 bytes, or almost 700 GiB. Not sure what you expected here. The command-line frontends will just fail the allocation and there's no harm. If you're linking to libjxl yourself, you can always just check the image dimensions before you attempt to allocate an output buffer.