lz4-java icon indicating copy to clipboard operation
lz4-java copied to clipboard

Data larger than 4194304 Bytes is not decompressed correctly

Open mgrundie-r7 opened this issue 2 years ago • 2 comments
trafficstars

    Random random = new Random();
    data = new byte[4194304];
    random.nextBytes(data);

    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

    LZ4FrameOutputStream lz4FrameOutputStream = new LZ4FrameOutputStream(byteArrayOutputStream);
    lz4FrameOutputStream.write(data);
    lz4FrameOutputStream.close();

    byte[] compressedBytes = byteArrayOutputStream.toByteArray();
    LZ4FrameInputStream compressedInputStream = new LZ4FrameInputStream(new ByteArrayInputStream(compressedBytes));

    byte[] decompressedData = new byte[data.length];
    compressedInputStream.read(decompressedData);
    compressedInputStream.close();

    assertTrue(Arrays.equals(data, decompressedData));

The above assertion passes. If you update the code to data = new byte[4194305]; then the assertion fails and the 4194305th element of decompressedData (decompressedData[4194304]) will equal 0

mgrundie-r7 avatar Jan 17 '23 15:01 mgrundie-r7

That's because you are decompressing only one block (you should call compressedInputStream .read() more than once).
Instead of:

byte[] decompressedData = new byte[data.length];
compressedInputStream.read(decompressedData);

Try:

byte[] decompressedData = compressedInputStream.readAllBytes();

petoncle avatar Feb 13 '23 23:02 petoncle

I can't control what the callers do and the callers have no specific knowledge which type of InputStream this is. I ended up writing an adapter that extends InputStream and delegates to LZ4FrameInputStream to add some reset capability and to chunk read when read is asked to read > 4194304

mgrundie-r7 avatar Feb 14 '23 08:02 mgrundie-r7