go-sync icon indicating copy to clipboard operation
go-sync copied to clipboard

Possible to stream files through an xz algorithim?

Open fire opened this issue 10 years ago • 6 comments

It may be possible to stream a part of the transfer through an xz algorithm such as https://bitbucket.org/Rawr/goxz/.

fire avatar Jan 25 '15 00:01 fire

There is some stuff missing for support of compressed blocks at the moment - particularly the block source implementation would need to translate a request for a set of blocks into a correct range in the compressed source.

Redundancy avatar Jan 27 '15 16:01 Redundancy

I'm working on some changes that should help to enable this by pulling the block source base lookup of the start and end of a block into an interface (which will allow compressed ranges). It would still need implementation of the compression, some extraction in the http requester and the range lookup, but that ought to enable block-based compression.

Redundancy avatar Feb 02 '15 12:02 Redundancy

https://stackoverflow.com/questions/28249782/is-it-possible-to-extract-a-tar-xz-package-in-golang/31491033#31491033

This is a pure golang implementation of xz.

fire avatar Jul 20 '15 04:07 fire

I'll hopefully have a look at compression soon

Redundancy avatar Jul 20 '15 17:07 Redundancy

Any updates?

fire avatar Oct 28 '15 09:10 fire

Sorry, I haven't had much time for this, and currently I can't see when I will in the next few months. Perhaps I'll manage to squeeze some time in somehow.

Redundancy avatar Oct 30 '15 15:10 Redundancy