Unbounded malloc DoS in PNM loading code

[cleeus]

The PNM parsing code reads the width and height fields and will allocate an appropriate CImg instance. Since this is done before reading the data and without a size check on the input file, even a very short PNM file can cause a huge allocation. Especially on 64bit platforms, the PNM loading code will allocate huge amounts of memory when supplied with huge W and H values, e.g. this PNM input will allocate 16GB and spend a lot of time in the parsing loop: P2 131072 131072 255

This can be considered a DoS since an attacker has to only supply a very short amount of data to case a huge ressource usage.

[cleeus]

First potential fix in https://github.com/dtschump/CImg/pull/296

[cleeus]

I'm unsure if this is the best fix possible. I can see an alternative fix but am unsure if this is good either: Instead of outright rejecting large dimensions, test if PNM file size roughly matches the number of expected pixels. Then reject PNM files that cannot possibly contain the claimed number of pixels. This would not work for pipe FILE* structes that cannot seek