neon icon indicating copy to clipboard operation
neon copied to clipboard

Add support for reading and writing compressed blobs

Open arpad-m opened this issue 2 months ago • 1 comments

Add support for reading and writing zstd-compressed blobs for use in image layer generation, but maybe one day useful also for delta layers. The reading of them is unconditional while the writing is controlled by the image_compression config variable allowing for experiments.

For the on-disk format, we re-use some of the bitpatterns we currently keep reserved for blobs larger than 256 MiB. This assumes that we have never ever written any such large blobs to image layers.

After the preparation in #7852, we now are unable to read blobs with a size larger than 256 MiB (or write them).

TODO:

  • ~~Maybe introduce a new version so that we give better errors should we encounter legacy image layers with such large blobs. This is to insure us in the case the assumption above is wrong, so there is > 256MiB large images.~~ eventually chosen against as image layers and delta layers have different ways of storing the version number.

A non-goal of this PR is to come up with good heuristics of when to compress a bitpattern. This is left for future work.

Parts of the PR were inspired by #7091.

cc #7879

Part of #5431

arpad-m avatar Jun 19 '24 00:06 arpad-m