Integer overflow in `stbi__convert_8_to_16`

[JarLob]

stbi__malloc in stbi__convert_8_to_16 [1] may overflow. However for successful exploitation img_len must be bigger than zero [2]. Any big enough img_len multiplied by 2 and casted to size_t on a 64 bit platform results in an unsigned number not smaller than the img_len.

static stbi__uint16 *stbi__convert_8_to_16(stbi_uc *orig, int w, int h, int channels)
{
   int i;
   int img_len = w * h * channels;
   stbi__uint16 *enlarged;

   enlarged = (stbi__uint16 *) stbi__malloc(img_len*2); // [1] int overflow
   if (enlarged == NULL) return (stbi__uint16 *) stbi__errpuc("outofmem", "Out of memory");

   for (i = 0; i < img_len; ++i) // [2]
      enlarged[i] = (stbi__uint16)((orig[i] << 8) + orig[i]); // replicate to high and low byte, maps 0->0, 255->0xffff

   STBI_FREE(orig);
   return enlarged;
}

Impact

It doesn't look like a potential security issue, but the signed integer overflow behavior is undefined according to C/C++ standard.

Resources

To reproduce the issue in stbi__vertical_flip_slices:

1. Make UBSAN build of the following program:

#include <stdint.h>
#define STB_IMAGE_IMPLEMENTATION
#include "../stb_image.h"

int main(int argc, char* argv[])
{
    const uint8_t data[] = {0xff,0xd8,0xff,0xc2,0x00,0x0b,0x08,0x11,0x01,0xff,
                            0x03,0x01,0x11,0x11};
    size_t size = sizeof(data);

    int x, y, channels;
    stbi_us *img = stbi_load_16_from_memory(data, size, &x, &y, &channels, 4);
    stbi_image_free(img);
    return 0;
}

2. Set breakpoint at line 1210 in stbi__convert_8_to_16 and run the program to hit the overflow.

/src/stb/tests/../stb_image.h:1210:52: runtime error: signed integer overflow: 1136707596 * 2 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /src/stb/tests/../stb_image.h:1210:52 in

[ericoporto]

@JarLob could add somewhere like in the title of these or the PRs WHAT stb libraries you are contributing the code/issue to? I only use stb_vorbis from here (and indirectly through SDL_Sound) and am having a hard time differentiating from all issues you opened - in the same day!