Backing store not supported

[strarsis]

After upgrading mozjpeg, compressing some JPEG images fails with error:

Command failed: [...]/mozjpeg/vendor/cjpeg -quality 95 -maxmemory 4096
Backing store not supported

[kornelski]

The executable you've got has been built incorrectly, without memory destination feature required by MozJPEG.

[strarsis]

@zvezdochiot: Why the thumbs down? This issue is qualified. Please don't just thumb down an issue.

[zvezdochiot]

And leave it there: https://github.com/imagemin/mozjpeg-bin/issues/49. But not here.

[strarsis]

@zvezdochiot: Well, now we found the reason for the error.

[strarsis]

@zvezdochiot, @kornelski: So after further diving into the code of mozjpeg I eventually found the underlying issue for this error. It is caused by code in the mozjpeg source, not by a particular build process or configuration for its distributed binaries.

mozjpeg calls jpeg_mem_available, when the returned numbers are too small for the given input file, it calls jpeg_open_backing_store, which always causes an exception, as jpeg_mem_available isn't supported anymore and the current workaround in mozjpeg consists of jpeg_mem_available always returning a large enough number of available memory amount for any input image size. But in my WSL 1 / 2 environment this isn't the case apparently, hence the returned available memory isn't sufficient and mozjpeg tries to use the now unsupported backing store feature.

Could this be relevant?

The libjpeg-turbo memory manager will now honor the max_memory_to_use structure member in jpeg_memory_mgr, which can be set to the maximum amount of memory (in bytes) that libjpeg-turbo should use during decompression or multi-pass (including progressive) compression. This limit can also be set using the JPEGMEM environment variable or using the -maxmemory switch in cjpeg/djpeg/jpegtran (refer to the respective man pages for more details.) -- Changelog entry for release 1.5.2

And indeed, mozjpeg invokes its vendored cjpeg binary with -maxmemory:

[...]/node_modules/mozjpeg/vendor/cjpeg -quality 95 -maxmemory 4096 large-enough-example.jpg > output.jpg

An input JPEG of about 932K is already sufficient to cause this error.

After dramatically increasing the maxmemory parameter, the error goes away as mozjpeg doesn't deem the allowed memory to be insufficient for compressing the JPEG input image.

Two issues in mozjpeg need to be addressed:

1. mozjpeg should throw a sensible error, that not enough memory is available, instead of trying to use a non-existing feature (backing store) as an (invalid) fallback that causes a confusing error on its own.
2. For a ~1MB large JPEG input file, maxmemory 4096 ^= 4000.000 bytes = 4 Megabytes (-maxmemory "Value is in thousands of bytes" (without a postfix like MB) according to the mozjpeg README) of memory seem to be insufficient to mozjpeg, this appears to be a miscalcuation. maxmemory had to be set to at least 13000 ^= 13.000.000 bytes = 13 Terabytes(!) to make it pass the current mozjpeg size checks. https://github.com/mozilla/mozjpeg/blob/d23e3fc58613bc3f0aa395a8c73a2b1e7dae9e25/usage.txt#L211-L213 Or is the documentation incorrect and for maxmemory a number of (singular) bytes (not thousands of bytes) is expected?

[strarsis]

@zvezdochiot: Would it be possible for you to look into this issue?