lz4-java
lz4-java copied to clipboard
lz4
Warning should be given first thing in the GitHub page that this library is not interoperable with other LZ4 tooling
As the title suggests. A warning is given on the LZ4 wiki page that this library does not conform the LZ4 standard and will not work with standard tooling. This looks to be a common issue (#184, #194 ) with people attempting to decompress files created with this library in other languages (or vice versa) and failing. A deviation from the standard should be mentioned, especially as this repo is under the lz4 account.
Is it true, is there some version that produces output not interoperable with previous versions of other tools too?
I see that this is among the custom (not interoperable) implementations in https://lz4.org/ homepage Because
They use the block compression format, but add their own frame / header logic (or none at all) Consequently, they are not interoperable with LZ4 command line utility, nor (generally) between themselves.
just it's unexpected because they come from the same official GitHub account.
It’s not about older versions of other tools, it’s all down to the block format. For our use case we ended up just switching to the Apache Java library which uses the Frame format instead, now I can use the lz4 cli to work with the files. In case of disaster I need access to these files from the cli, I don’t have time to use any Java script in the middle, if it’s not present on the server then it’s useless to me.
I do agree with you though, it is very much unexpected this is not mentioned at all on this project’s readme, it should be made clearer.
On Sat, 17 Aug 2024 at 00:15, Emanuele Sabellico @.***> wrote:
I see that this is among the custom (not interoperable) implementations in https://lz4.org/ homepage Because
They use the block https://github.com/lz4/lz4/blob/master/doc/lz4_Block_format.md compression format, but add their own frame / header logic (or none at all) Consequently, they are not interoperable with LZ4 command line utility, nor (generally) between themselves.
just it's unexpected because they come from the same official GitHub account.
— Reply to this email directly, view it on GitHub https://github.com/lz4/lz4-java/issues/222#issuecomment-2294433947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVLI3DW7TIQWUYUT6NSF73ZR2B2PAVCNFSM6AAAAABLACE54OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUGQZTGOJUG4 . You are receiving this because you authored the thread.Message ID: @.***>
If you only ever intend on using Java to work with the compress files this project will work for you. If you want to use the cli too then look elsewhere.
On Sat, 17 Aug 2024, 00:40 Andrei Vasile, @.***> wrote:
It’s not about older versions of other tools, it’s all down to the block format. For our use case we ended up just switching to the Apache Java library which uses the Frame format instead, now I can use the lz4 cli to work with the files. In case of disaster I need access to these files from the cli, I don’t have time to use any Java script in the middle, if it’s not present on the server then it’s useless to me.
I do agree with you though, it is very much unexpected this is not mentioned at all on this project’s readme, it should be made clearer.
On Sat, 17 Aug 2024 at 00:15, Emanuele Sabellico @.***> wrote:
I see that this is among the custom (not interoperable) implementations in https://lz4.org/ homepage Because
They use the block https://github.com/lz4/lz4/blob/master/doc/lz4_Block_format.md compression format, but add their own frame / header logic (or none at all) Consequently, they are not interoperable with LZ4 command line utility, nor (generally) between themselves.
just it's unexpected because they come from the same official GitHub account.
— Reply to this email directly, view it on GitHub https://github.com/lz4/lz4-java/issues/222#issuecomment-2294433947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVLI3DW7TIQWUYUT6NSF73ZR2B2PAVCNFSM6AAAAABLACE54OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUGQZTGOJUG4 . You are receiving this because you authored the thread.Message ID: @.***>
I was checking these comments and seems like the official frame format was implemented in 1.4.0
https://github.com/lz4/lz4-java/issues/21#issuecomment-22927686 https://github.com/lz4/lz4-java/issues/21#issuecomment-422155543
Looking at the documentation,
this java implementation seems to offer 2 sets of API,
one is LZ4Compressor/LZ4SafeDecompressor,
the other is LZ4FrameOutputStream/LZ4FrameInputStream.
The LZ4Frame API should be directly compatible with lz4 CLI.
The other API likely builds on top of the inner lz4 block format, and may or may not add some metadata of its own, making it incompatible with the lz4 CLI and other interoperable LZ4 decoders.
Maybe this distinction should be better documented.
I didn't catch that, that's a very useful find. Thank you for finding it both. I stand corrected
On Fri, 23 Aug 2024, 23:42 Yann Collet, @.***> wrote:
Looking at the documentation, this java implementation seems to offer 2 sets of API, one is LZ4Compressor/LZ4SafeDecompressor, the other is LZ4FrameOutputStream/LZ4FrameInputStream.
The LZ4Frame API should be directly compatible with lz4 CLI.
The other API likely builds on top of the inner lz4 block format, and may or may not add some metadata of its own, making it incompatible with lz4.
Maybe this distinction should be better documented.
— Reply to this email directly, view it on GitHub https://github.com/lz4/lz4-java/issues/222#issuecomment-2307894035, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVLI3D4FFQ5FMBNSBOIAHDZS63EBAVCNFSM6AAAAABLACE54OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXHA4TIMBTGU . You are receiving this because you authored the thread.Message ID: @.***>