apache
Add zstd compression/decompression IO Stream
zstd decompression/compression should be added a standard pekko IO stream as its far superior to gzip/lzma in every way. The API design should be the same as it is for gzip.
Doing this requires using https://github.com/luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).
We pre-trained the dictionary and delivered it to the client. If you want to leverage it.
We pre-trained the dictionary and delivered it to the client. If you want to leverage it.
I will add an API for compressing/decompressing via dictionaries as well, ill see if adding an API to create dictionaries is warranted (as you can just use zstd-jni directly for this)
Doing this requires using luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).
There is at https://github.com/airlift/aircompressor
(I also wrote a very high-level, pure implementation in Scala using scodec but it's not optimized and ~ 100-1000x slower than native at https://github.com/jrudolph/decoders/blob/main/src/main/scala/net/virtualvoid/codecs/zstd/Zstd.scala)
Doing this requires using luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).
There is at https://github.com/airlift/aircompressor
According to their readme
This library requires a Java 22+ virtual machine containing the sun.misc.Unsafe interface running on a little endian platform.
So I don't think it can be used as Pekko 2.0.x supports JVM 1.7 or older. Another alternative would be to only release an interface for zstd compression/decompression and there is a separate pekko artifact for the implementation which can be zstd-jni or airlift (or anything else) but that this to me is a bit of overengineering but I can be convinced otherwise.
FYI, Netty 5 will require JDK 25:)
Yup, but would be a Pekko 3.0.0 before we can use it, and at that time we can also consider swapping out the underlying zstd implementation for a better one if it exists (which we can do since a bump in major version allows breaking changes)