[BUG, CVE-2019-8396, HDFFV-10712] Heap Buffer Overflow in H5O__pline_decode()

[e4t]

Description CVE-2019-8396 provides a reproducer that causes a heap buffer overflow in H5O__pline_decode(). The issue is still seen in hdf5 version 1.13.3. The cause of this is an incomplete bounds checking in said function as only the the address of the filter descriptions is checked against the bounds. If the overflow happens before, because there isn't even enough space for the filter pipeline header, an overflow will be triggered. Once this overflow condition is handled properly, the following overflow in H5O__layout_decode no longer occurs. On version 1.13 the SEGV seen in the link above no longer occurs. The heap overflow itself appears to be harmless from a security perspective. This may not be true for later memory accesses (possibly writes) based on inconsistent/invalid data read. The overflow can be seen both with valgrind and the gcc address sanitizer (make CFLAGS="-g -fsanitize=address" CXXFLAGS="-g -fsanitize=address" FCFLAGS="-g -fsanitize=address")

Expected behavior Detect invalid hdf5 data properly and trigger the appropriate error handler. Handle buffer overflow condition properly to avoid possible vulnerabilities thru manipulated hdf5 data files brought in circulation by bad actors.

Platform

• HDF5 version 1.13.3
• openSUSE Leap 15.4 / SLE 15.4 / openSUSE Tumbleweed
• gcc 7 / gcc 11
• autotools
• configure --enable-fortran --enable-unsupported --enable-hl --enable-shared --enable-threadsafe --enable-build-mode=production --enable-cxx --with-pthread