Support configurable constraints on image size during parsing

[tdsmith]

When handling potentially hostile user input, it would be helpful if libgd provided an API to constrain the size of the maximum potential memory allocation to substantially less than INT_MAX bytes. This could take the form of constraints on image dimensions or the pixel count. Otherwise, it's possible for a very small file to specify very large dimensions in its header and have the libgd process OOM killed on constrained systems when it tries to allocate a buffer for it.

I noticed this while trying to implement fuzzers for libgd; to avoid the fuzzer being OOM killed I have to do my own parsing of the file formats before passing the buffers on to libgd, or override INT_MAX in gd_security.c. It would be unrealistic to expect clients to do this in their web applications but a similar concern is present when processing user-presented data.

This is related to #122, which argues that the INT_MAX constraint is actually too small in some cases.

Thanks!

[cmb69]

Maybe implementing #335 would be a (partial) solution? (I mean that one could constrain the amount of memory consumed by libgd this way.)

[vapier]

i'm not sure capping all memory allocations is the right route for fuzzers. there's two scenarios here:

• fuzzed input generates a valid image with super big dimensions
• fuzzed input generates an invalid image which causes gd to make a large/bad allocation due to some bad integer math

if we just threw out/ignored all large allocations, we'd harm our coverage.

#122 would handle this by virtue of gd paging images, but it seems pretty unlikely we'd come up with a solution for that any time soon.

iiuc, Google's oss-fuzz utilizes libfuzzer which means we need to specially compile all of libgd, so we could add some ifdef logic in some core paths to handle this.

[tdsmith]

Hmm. I guess I would have reasoned that a large, valid image is likely to behave like a small, valid image. Anecdotally, fuzzers seem to like generating examples of the latter case, particularly since the size of the fuzzing data is constrained; I'm optimistic about the ability of a fuzzer to exercise cases where w*h > size(buf) even with constraints on w*h. Is that what you were thinking about? It should be possible to collect empirical coverage data to support my wild claims. :)

[cmb69]

iiuc, Google's oss-fuzz utilizes libfuzzer which means we need to specially compile all of libgd, so we could add some ifdef logic in some core paths to handle this.

It would be sufficient to set constrains in gdImageCreate() and gdImageCreateTrueColor(), wouldn't it?