nydus icon indicating copy to clipboard operation
nydus copied to clipboard

Implement compression dictionary (aka training mode) for chunk data compression & decompression in blob file

Open liubogithub opened this issue 3 years ago • 0 comments

Per Lz4 project page

LZ4 is also compatible with dictionary compression, both at API and CLI levels. It can ingest any input file as dictionary, though only the final 64KB are used. This capability can be combined with the Zstandard Dictionary Builder, in order to drastically improve compression performance on small files.

Since we're using crate 'lz4-sys' for compressing & decompressing chunk data and more important, chunk data is up to 1MB which is considered as small data set for lz4 alike, it is likely that nydus can benefit a lot from applying compression dictionary.

Lz4 has offered some support at API level, we need to look into how to enable it for rafs code.

liubogithub avatar Dec 13 '21 18:12 liubogithub