neon
neon copied to clipboard
Add support for reading and writing compressed blobs
Add support for reading and writing zstd-compressed blobs for use in image layer generation, but maybe one day useful also for delta layers. The reading of them is unconditional while the writing is controlled by the image_compression
config variable allowing for experiments.
For the on-disk format, we re-use some of the bitpatterns we currently keep reserved for blobs larger than 256 MiB. This assumes that we have never ever written any such large blobs to image layers.
After the preparation in #7852, we now are unable to read blobs with a size larger than 256 MiB (or write them).
TODO:
- ~~Maybe introduce a new version so that we give better errors should we encounter legacy image layers with such large blobs. This is to insure us in the case the assumption above is wrong, so there is > 256MiB large images.~~ eventually chosen against as image layers and delta layers have different ways of storing the version number.
A non-goal of this PR is to come up with good heuristics of when to compress a bitpattern. This is left for future work.
Parts of the PR were inspired by #7091.
cc #7879
Part of #5431