WebP Crash: partition 0 overflow (> 512K) with method 0/1 on 60MP JPG in Release Mode

[PatJDooley]

Magick.NET version

14.4.0 Magick.NET-Q16-AnyCPU

Environment (Operating system, version and so on)

Windows 11, Visual Studio 2022, 64-bit,

Description

Converting a 60MP JPG (9504x6336px, 34MB) to WebP with MagickFormat.Webp, "method", "0" or "1", crashes in release mode but works in debug. "method", "2"+ are stable.

Debugging in Release mode gave this error:

HResult=0x80131500 Message=One or more errors occurred. Source=<Cannot evaluate the exception source> StackTrace: <Cannot evaluate the exception stack trace>

This exception was originally thrown at this call stack: [External Code]

Inner Exception 1: MagickCorruptImageErrorException: partition 0 overflow (> 512K) `D:\2024\April\PJD09269.webp' @ error/webp.c/WriteSingleWEBPImage/893

Steps to Reproduce

Steps to Reproduce:

1. Load 9504x6336px JPG (34MB) with MagickImage.
2. Set image.Settings.SetDefine(MagickFormat.Webp, "method", "0"), quality 80.
3. Write to WebP in release build (VS Community, .NET 6/8).
4. Crashes after processing around 216 similar JPG files. .

Error:

Sorry, File won't upload. I can email it.

Images

[dlemstra]

I will update the issue template to make this more clear but you can zip it in multiple files with a size limit per zip file. But it also looks like I need 216 files to reproduce this? I would advise you to create a repository on GitHub that demonstrates the issue. I don't have enough information to reproduce your issue.

[PatJDooley]

I moved the image(s) causing the issues to a separate folder and got these errors. Because of parallelism, different threads may hit different images, so two different images have caused issues. I've broken them down into 10mb zip files.

ImageMagickTest001.zip ImageMagickTest002.zip ImageMagickTest003.zip ImageMagickTest004.zip ImageMagickTest005.zip ImageMagickTest006.zip ImageMagickTest007.zip

"0"

ImageMagick.MagickCorruptImageErrorException HResult=0x80131500 Message=partition 0 overflow (> 512K) C:\Temp\2024\April\PJD09294.WebP' @ error/webp.c/WriteSingleWEBPImage/893 Source=Magick.NET-Q16-AnyCPU StackTrace: at ImageMagick.MagickImage.NativeMagickImage.WriteFile(IMagickSettings1 settings) at CRIMP.MainWindow.<>c__DisplayClass37_2.<ConvertResizeDelete>b__4(String file) in D:\My Software Projects\Dot Net\CRIMP\MainWindow.xaml.cs:line 312 at System.Threading.Tasks.Parallel.<>c__DisplayClass43_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

"1"

ImageMagick.MagickCorruptImageErrorException HResult=0x80131500 Message=partition 0 overflow (> 512K) C:\Temp\2024\April\PJD09269.WebP' @ error/webp.c/WriteSingleWEBPImage/893 Source=Magick.NET-Q16-AnyCPU StackTrace: at ImageMagick.MagickImage.NativeMagickImage.WriteFile(IMagickSettings1 settings) at CRIMP.MainWindow.<>c__DisplayClass37_2.<ConvertResizeDelete>b__4(String file) in D:\My Software Projects\Dot Net\CRIMP\MainWindow.xaml.cs:line 312 at System.Threading.Tasks.Parallel.<>c__DisplayClass43_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

[dlemstra]

Can you also reproduce this with the latest release that was published today?

[PatJDooley]

Yes. I shall test that today.

BTW, ImageMagick is really fast at "2". I converted 5244 files totaling 42GB of JPG space to 16GB of webP space in 5 minutes. At "6", it took 22 minutes resulting in 15GB of webP space.

[PatJDooley]

Same error using 14.5.0.

ImageMagick.MagickCorruptImageErrorException HResult=0x80131500 Message=partition 0 overflow (> 512K) C:\Temp\ImageMagickTest\PJD09294.WebP' @ error/webp.c/WriteSingleWEBPImage/893 Source=Magick.NET-Q16-AnyCPU StackTrace: at ImageMagick.MagickExceptionHelper.Check(IntPtr exception) at ImageMagick.NativeHelper.CheckException(IntPtr exception) at ImageMagick.MagickImage.NativeMagickImage.WriteFile(IMagickSettings1 settings) at ImageMagick.MagickImage.Write(String fileName) at CRIMP.MainWindow.<>c__DisplayClass37_2.<ConvertResizeDelete>b__4(String file) in D:\My Software Projects\Dot Net\CRIMP\MainWindow.xaml.cs:line 313 at System.Threading.Tasks.Parallel.<>c__DisplayClass43_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

[dlemstra]

It looks like you are running into a limitation of the webp format (https://developers.google.com/speed/webp/docs/cwebp)

Degrade quality by limiting the number of bits used by some macroblocks. Range is 0 (no degradation, the default) to 100 (full degradation). Useful values are usually around 30-70 for moderately large images. In the VP8 format, the so-called control partition has a limit of 512k and is used to store the following information: whether the macroblock is skipped, which segment it belongs to, whether it is coded as intra 4x4 or intra 16x16 mode, and finally the prediction modes to use for each of the sub-blocks. For a very large image, 512k only leaves room for a few bits per 16x16 macroblock. The absolute minimum is 4 bits per macroblock. Skip, segment, and mode information can use up almost all these 4 bits (although the case is unlikely), which is problematic for very large images. The partition_limit factor controls how frequently the most bit-costly mode (intra 4x4) will be used. This is useful in case the 512k limit is reached and the following message is displayed: Error code: 6 (PARTITION0_OVERFLOW: Partition #0 is too big to fit 512k). If using -partition_limit is not enough to meet the 512k constraint, one should use less segments in order to save more header bits per macroblock. See the -segments option. Note the -m and -q options also influence the encoder's decisions and ability to hit this limit.

I suspect you will need to go through these docs to find the best options for your situation.

[PatJDooley]

Thank you for that information.. It is good to know that the root cause is an inherent limitation of the webP format and those limitations can be avoided by setting the parameters appropriately.