fury icon indicating copy to clipboard operation
fury copied to clipboard

[Java] Implement zstd based compressor for type metadata

Open chaokunyang opened this issue 1 year ago • 2 comments

Is your feature request related to a problem? Please describe.

In https://github.com/apache/incubator-fury/pull/1663, we add a MetaCompressor interface and deflater based implementation.

Zstd has better compression rate and can learn from train corpu, we should support using zstd as meta compressor

Describe the solution you'd like

Add a new java maven module, impement zstd based meta compressor.

Additional context

#1663

chaokunyang avatar May 30 '24 09:05 chaokunyang

@chaokunyang Hi, I would like to try this. But i have question first, do i need create new maven module for this? It looks like just add new Class in fury-core?

wangjie-fourth avatar Jun 05 '24 08:06 wangjie-fourth

If we add a new class directly into fury-core, we must aovid introducing a new jar dependency. MethodHandle or reflection is feasible. If we don't use reflection, we may need to add a new maven module

chaokunyang avatar Jun 05 '24 15:06 chaokunyang