heap_buffer_overflow_in_transformDataUnit

[Cvjark]

Hi, in the lastest version of this code [ ps: commit id ffaf11c] I found something unusual.

crash sample

8id64_heap_buffer_overflow_in_transformDataUnit.zip

command to reproduce

./pdftops -q [crash sample] /dev/null

crash detail

=================================================================
==115877==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6200000080e0 at pc 0x000000756136 bp 0x7fff10b0da30 sp 0x7fff10b0da28
READ of size 2 at 0x6200000080e0 thread T0
    #0 0x756135 in DCTStream::transformDataUnit(unsigned short*, int*, unsigned char*) /home/bupt/Desktop/xpdf/xpdf/Stream.cc:2968:17
    #1 0x748741 in DCTStream::decodeImage() /home/bupt/Desktop/xpdf/xpdf/Stream.cc:2835:6
    #2 0x7402bb in DCTStream::reset() /home/bupt/Desktop/xpdf/xpdf/Stream.cc:2261:5
    #3 0x68912e in Object::streamReset() /home/bupt/Desktop/xpdf/xpdf/./Object.h:282:13
    #4 0x68912e in Lexer::Lexer(XRef*, Object*) /home/bupt/Desktop/xpdf/xpdf/Lexer.cc:74:12
    #5 0x581714 in Gfx::display(Object*, int) /home/bupt/Desktop/xpdf/xpdf/Gfx.cc:641:33
    #6 0x6a76a1 in Page::displaySlice(OutputDev*, double, double, int, int, int, int, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/Page.cc:360:10
    #7 0x6d5f6e in PSOutputDev::checkPageSlice(Page*, double, double, int, int, int, int, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/PSOutputDev.cc:3276:11
    #8 0x6a7172 in Page::displaySlice(OutputDev*, double, double, int, int, int, int, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/Page.cc:328:13
    #9 0x6a6f81 in Page::display(OutputDev*, double, double, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/Page.cc:308:3
    #10 0x6af9b4 in PDFDoc::displayPage(OutputDev*, int, double, double, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/PDFDoc.cc:384:27
    #11 0x6af9b4 in PDFDoc::displayPages(OutputDev*, int, int, double, double, int, int, int, int, int (*)(void*), void*) /home/bupt/Desktop/xpdf/xpdf/PDFDoc.cc:397:5
    #12 0x796d81 in main /home/bupt/Desktop/xpdf/xpdf/pdftops.cc:342:10
    #13 0x7f57efb1ec86 in __libc_start_main /build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:310
    #14 0x41d5d9 in _start (/home/bupt/Desktop/xpdf/xpdf/pdftops+0x41d5d9)

Address 0x6200000080e0 is a wild pointer.
SUMMARY: AddressSanitizer: heap-buffer-overflow /home/bupt/Desktop/xpdf/xpdf/Stream.cc:2968:17 in DCTStream::transformDataUnit(unsigned short*, int*, unsigned char*)
Shadow bytes around the buggy address:
  0x0c407fff8fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff8fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff8fe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff8ff0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff9000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c407fff9010: fa fa fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa
  0x0c407fff9020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff9030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff9040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff9050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c407fff9060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==115877==ABORTING