FR: Support Block Dependence

[parvez]

I am getting following error for some lz4 files.

Error: Invalid data block: 12 @579083
    at Decoder.emit_Error (src/node_modules/lz4/lib/decoder_stream.js:64:22)
    at Decoder.uncompress_DataBlock (src/node_modules/lz4/lib/decoder_stream.js:246:9)
    at Decoder._main (src/node_modules/lz4/lib/decoder_stream.js:317:25)
    at Decoder._transform (src/node_modules/lz4/lib/decoder_stream.js:60:7)
    at Decoder.Transform._read (_stream_transform.js:167:10)
    at Decoder.Transform._write (_stream_transform.js:155:12)
    at doWrite (_stream_writable.js:301:12)
    at writeOrBuffer (_stream_writable.js:287:5)
    at Decoder.Writable.write (_stream_writable.js:215:11)
    at IncomingMessage.ondata (_stream_readable.js:536:20)

I added some debug code in "Decoder.prototype.uncompress_DataBlock" function in decoder_stream.json

this.descriptor.blockMaxSize 262144 , decodedSize262144 this.descriptor.blockMaxSize 262144, decodedSize -4 <<< This is where it errors out

[pierrec]

How did you compress the files? (i.e. via node-lz4 or some other tool?) Could you provide a small sample file that reproduces the issue?

[parvez]

Found the issue was with block independence flag. Will you be adding support for block dependence flag in future release?

Non working lz4 as block independence is false

node ./node_modules/lz4/examples/file_uncompressSync.js original-non-working.lz4
 
this.descriptor { blockIndependence: false,
  blockChecksum: false,
  blockMaxSize: 262144,
  streamSize: false,
  streamChecksum: false,
  dict: false,
  dictId: 0 }

Working lz4 as block independence is true

node ./node_modules/lz4/examples/file_uncompressSync.js my-working-1.lz4

this.descriptor { blockIndependence: true,
  blockChecksum: false,
  blockMaxSize: 4194304,
  streamSize: false,
  streamChecksum: true,
  dict: false,
  dictId: 0 }

[pierrec]

The block dependency compression is indeed not implemented in the js version. It would be nice to have it, even though the few samples I tried it on (using the Go version) didnt bring any benefit. I dont have time right now to do the work but if you feel like it I would consider a pull request! Or maybe for the time being, just emit an error when the flag is set.

[parvez]

Thanks Pierre for the follow up! Hopefully someone can help with a PR for this. 👍

[alexfernandez]

@pierrec I am under the impression that the C version doesn't support block dependence either, is this so? sample.lz4 cannot be decoded with file.js, with or without useJS:true commented; however it works with unlz4 under Debian.

Note: the enclosed file sample.gz can be decompressed using:

$ tar -xzf sample.gz

sample.gz

[khelkun]

@pierrec @parvez I encounter the same error with some files compressed with the stream interface of this K4os.Compression.LZ4 C# library so I susspect the issue is also about "block dependence". When you talk about "block dependence" is there any chance it's related to the ChainBlocks flag in the stream compression settings of K4os.Compression.LZ4 ?

[khelkun]

@pierrec Sorry to up my previous question but I'd need to figure out why node-lz4 fails to uncompress some of my lz4 files (not all of them).

So far I've used lz4js because it does decode my lz4 files compressed with K4os.Compression.LZ4 C# library.

However lz4js has no stream support and I wish to use streams because I have some big lz4 files to uncompress.