protocol
protocol copied to clipboard
Spinoco
[WIP] add LZ4 kafka support
I would like to add LZ4 support as our production topics uses this compression.
The tests don't pass.
I am pretty sure using LZ4FrameInputStream and LZ4FrameOutputStream is the issue here the lz4 java library does not supply a standard InputStream / OutputStream
This https://github.com/boy0001/FastAsyncWorldedit/blob/master/core/src/main/java/net/jpountz/lz4/LZ4OutputStream.java
looks more like it.
Not sure how to approach this, what do you think @pchlupacek ?
@dsebban thanks for this.
Regarding the tests, I am not sure how the LZ4 implementation work, but we have to somehow be able to take a byte of compressed data and uncompress it which is essentially ByteVector => Attempt[ByteVector] function. As such it has nothing to do whether this is implemented as InputStream/OutputStream algorithm or not.
One think we have to look carefully though is that stream won't be performing any thread blocking operations, which in case of OS/IS means usually no Input/Output at all.
From the signature of LZ4OutputStream I think that shall be doable, given supplied OutputStream won't be blocking, such as ByteBuffer one.
Given your changes I think they are generally ok, so I suspect the LZ4 implementation needs some tweaks to one used in kafka :-(. You have to see any play with it a bit, I am not expert over LZ4.
@dsebban also maybe it should be a worth to look at LZ4Compressor, perhaps that would give you chance to implement that vie standard fs2 Pull/pipe instead of relying on IS/OS types.
cool , thanks for the pointers , I will play a bit and try to work out a non blocking implementation of LZ4
@pchlupacek I ended up importing KafkaLZ4BlockOutputStream.java and KafkaLZ4BlockInputStream.java only in a java src folder from the kafka client, they tweaked it heavily, so reimplementing it seems like a lot of work.
Also v0 messages does not work, let me know how I should proceed and if this approach is not acceptable.
@dsebban cool. thanks for your work.
I need to check on licence things for the files you have imported. They are apache, and fs2-kafka is MIT so I hope they can be mixed.
I will review this in few days, as we expect release fs2-kafka with 1.0-M4.
@dsebban , @cleverdeveloper sorry for being lte on this. Somehow this was missed under my radar. I will take a look on this this week if thats ok ?
I did a temporary release of Kafka part of this repository meanwhile with LZ4 stuff included: https://github.com/evolution-gaming/kafka-protocol