go-lz4 icon indicating copy to clipboard operation
go-lz4 copied to clipboard

Published 20 hours ago • verified publisher icon bkaradzic

encode with io.Writer / decode with io.Reader

Open lachie opened this issue 10 years ago • 14 comments

I'd love to use this code, but my project involves compressing files up to gigabytes in size.

For that use case I really need a streaming version of the algorithm (i.e. using the io.Writer/io.Reader interfaces). I see there's one as part of the reference C implementation, but I really have no idea where to start with porting it to Go.

lachie avatar May 07 '14 06:05 lachie

The io.Reader/io.Writer interface was removed in https://github.com/bkaradzic/go-lz4/commit/5298eaf858a62a1a1195bebb8e280ffca025237d due to bugs and slowness.

I agree having a streaming interface would be useful. IIRC, the old C library faked streaming by buffering in (~65k?) blocks and compressing in batches. You could do something similar and construct your own framing, assuming you don't need compatibility with any other implementation.

I see there is now a more complete spec for an LZ4 streaming format: https://docs.google.com/document/d/1gZbUoLw5hRzJ5Q71oPRN6TO4cRMTZur60qip-TE7BhQ/edit . We should probably support this format.

dgryski avatar May 07 '14 07:05 dgryski

Looks like the streaming format isn't completely implementing in the C lz4 implementation: https://code.google.com/p/lz4/issues/detail?id=42

dgryski avatar May 07 '14 07:05 dgryski

ok no worries, that sounds like a good solution

lachie avatar May 08 '14 05:05 lachie

Leaving this open as a reminder to put back the streaming support.

dgryski avatar May 08 '14 06:05 dgryski

http://fastcompression.blogspot.fr/2014/05/streaming-api-for-lz4.html

dgryski avatar May 20 '14 09:05 dgryski

Any progress about steaming support ?

vtolstov avatar Jul 14 '14 05:07 vtolstov

Spec has been finalised. I'll add it nearer the top of my todo list.

dgryski avatar Jul 14 '14 05:07 dgryski

@dgryski hello, do you have any news about streaming support?

vtolstov avatar Oct 15 '14 05:10 vtolstov

ping...

vtolstov avatar Jan 20 '15 10:01 vtolstov

I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.

Sorry :(

dgryski avatar Jan 20 '15 10:01 dgryski

2015-01-20 13:39 GMT+03:00 Damian Gryski [email protected]:

I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.

Sorry :(

Ok. May be i'm try.

Vasiliy Tolstov, e-mail: [email protected] jabber: [email protected]

vtolstov avatar Jan 20 '15 10:01 vtolstov

ping...

Any progress so far?

Or does anyone has good working example for using blocks and compressing in batches?

xxorde avatar Sep 24 '16 20:09 xxorde

Some of the other Go lz4 ports have implemented the streaming interface: https://github.com/pierrec/lz4 for example

dgryski avatar Sep 24 '16 20:09 dgryski

Thank you for this hint. It was way harder to implement this myself than I thought.

xxorde avatar Sep 25 '16 08:09 xxorde