nydus
nydus copied to clipboard
Implement compression dictionary (aka training mode) for chunk data compression & decompression in blob file
Per Lz4 project page
LZ4 is also compatible with dictionary compression, both at API and CLI levels. It can ingest any input file as dictionary, though only the final 64KB are used. This capability can be combined with the Zstandard Dictionary Builder, in order to drastically improve compression performance on small files.
Since we're using crate 'lz4-sys' for compressing & decompressing chunk data and more important, chunk data is up to 1MB which is considered as small data set for lz4 alike, it is likely that nydus can benefit a lot from applying compression dictionary.
Lz4 has offered some support at API level, we need to look into how to enable it for rafs code.