Zstd(:chunked) work tracking checklist

[mtrmac]

• Zstd in general:
  • [ ] Switch to Zstd default, earlier than a Zstd:chunked default? Testing in https://github.com/containers/podman/pull/21571 ; https://github.com/containers/podman/pull/21793
  • [x] Not a bug: pull of gzip + push of zstd changes image ID due to triggering a conversion to OCI: https://github.com/containers/podman/issues/20611
  • [x] Pushes with Zstd compression prefer using schema1 manifests: https://github.com/containers/image/issues/2181 , https://github.com/containers/image/pull/2196
  • [x] Pushes with Zstd compression to schema-agnostic transports don’t trigger a conversion: https://github.com/containers/image/issues/2195 , https://github.com/containers/image/pull/2204- Zstd:chunked:
  • [x] Blob reuse is not restricted by possible manifest schema options: https://github.com/containers/image/issues/2205 , https://github.com/containers/image/pull/2213
  • [x] Uncompressed pushes of originally-compressed layers don’t correctly update MIME types (always a bug, but now breaks pushes of originally-Zstd to docker-archive: destinations): https://github.com/containers/image/issues/2182
• Zstd:chunked:
  • [ ] copy.Options.EnsureCompressionVariantsExist would never stop adding zstd:chunked variants: https://github.com/containers/image/issues/2318
  • [ ] See if the BlobCache needs updating for Zstd / Zstd:chunked: https://issues.redhat.com/browse/RUN-1124
  • [ ] The chunked manifest size has a hard-coded limit, and pulling large layers outright fails: https://github.com/containers/podman/issues/24885
  • [x] Exercise Podman tests with Zstd:chunked default: https://github.com/containers/podman/pull/21903
  • [x] “view ambiguity”: https://issues.redhat.com/browse/RHEL-66492 , https://github.com/containers/storage/pull/2155 , https://github.com/containers/image/pull/2613 , https://github.com/containers/podman/pull/25007
  • [x] Pulls with missing parent directories fail: https://issues.redhat.com/browse/RUN-2364: Fixed in https://github.com/containers/storage/pull/2194
  • [x] Reusing already-present data can be optimized: https://github.com/containers/storage/pull/1830, https://github.com/containers/image/pull/2299, https://github.com/containers/image/pull/2583
  • [x] Podman’s zstd:chunked-specific tests are not representative: https://github.com/containers/podman/pull/24686
  • [x] Pulls on VFS fail outright: https://github.com/containers/podman/issues/24308 , https://github.com/containers/storage/pull/2140
  • [x] expectedLayerDiffIDFlag seems to use mismatching types: https://github.com/containers/image/issues/2602 , https://github.com/containers/image/issues/2602
  • [x] Document zstd:chunked and encryption interaction: https://github.com/containers/common/issues/2117, https://github.com/containers/common/pull/2176, https://github.com/containers/skopeo/pull/2427, https://github.com/containers/buildah/pull/5759, https://github.com/containers/podman/pull/24113
  • [x] Propose iterateTarSplit to upstream tar-split: https://github.com/containers/storage/pull/2035#discussion_r1680130116 . Filed https://github.com/vbatts/tar-split/pull/71 , https://github.com/containers/storage/pull/2116 .
  • [x] blobPipelineDetectCompressionStep detects zstd:chunked as zstd, causing unnecessary recompression: around https://github.com/containers/image/pull/1980#discussion_r1388492171 , https://github.com/containers/image/pull/2487 .
  • [x] Pushes reusing chunked layers don’t create the required annotations, making the layers impossible to pull chunked: https://github.com/containers/image/pull/2183#pullrequestreview-1725228215 / https://github.com/containers/image/pull/2185 , https://github.com/containers/image/pull/2487
  • [x] zstd:chunked and layer encryption don’t make sense together: #2485
  • [x] Blocker? TOC data and tar-split may be ambiguous: https://github.com/containers/storage/issues/2014 , https://github.com/containers/storage/pull/2035
  • [x] c/storage AdditionalLayerStore seems to be used for TOC-identified layers, incorrectly: https://github.com/containers/storage/pull/795#issuecomment-2007355572 , https://github.com/containers/storage/pull/1924
  • [x] TarSplitChecksumKey not used in a layer ID: https://github.com/containers/storage/issues/1888 , https://github.com/containers/storage/pull/1902
  • [x] blobPipelineCompressionStep would trigger a recompression of zstd:chunked if the user asks for zstd (except that we don’t currently detect zstd:chunked): around https://github.com/containers/image/pull/1980#discussion_r1388493306 , https://github.com/containers/image/pull/2317
  • [x] The c/storage “binary footer” code path does not work: https://github.com/containers/storage/issues/1886
  • [x] Outstanding items from https://github.com/containers/storage/pull/1627#pullrequestreview-1600645581 : https://github.com/containers/storage/pull/1844
  • [x] Copies of chunked layers don’t mark the image as requiring Zstd support: https://github.com/containers/image/issues/2077 , #2302
  • [x] Chunked layers are visible in c/storage, and usable by other processes, while they are still being populated: https://github.com/containers/image/issues/1979, https://github.com/containers/storage/pull/1826 + #2301 .
  • [x] PutBlobPartial with a non-chunked input and conversion in c/storage enabled doesn’t work: https://github.com/containers/image/pull/2288#issuecomment-1933549444 , fixed there
  • [x] Pushes of chunked layers fail because the UncompressedDigest fields stores a TOC digest, and layer digest validation fails: Fixed in https://github.com/containers/image/pull/1980
  • [x] Pulls of chunked layers do not reuse locally-existing layers: Part of https://github.com/containers/image/pull/1980
  • [x] In c/storage, we don’t sufficiently differentiate layers pulled by blob digest vs. TOC digest, and reuse is unclear: Part of https://github.com/containers/image/pull/1980 , and https://github.com/containers/storage/pull/1627#discussion_r1309255199 . Fixed in #2288 .
  • [x] Pulls may reuse layers pulled by blob digests vs. TOC digests: Part of https://github.com/containers/image/pull/1980 . Fixed in #2288 .
  • [x] Image deduplication contains a (required!) sanity check that TopLayer matches, but that might not be true if layers pulled by blob vs. TOC digests have different IDs: Part of https://github.com/containers/image/pull/1980 . Fixed in #2288 .
  • [x] Reuse of local layers by TOC may trigger inconsistent metadata updates: #2294
  • [x] A pull by TOC + conversion to non-OCI might parse data incorrectly: #2295

[giuseppe]

@flouthoc PTAL

[rhatdan]

Should

Pushes reusing chunked layers don’t create the required annotations, making the layers impossible to pull chunked: https://github.com/containers/image/pull/2183#pullrequestreview-1725228215 / https://github.com/containers/image/pull/2185

Be closed?

[cgwalters]

For the record I think it was me that recently added a zstd:chunked label on issues - there's some more stuff there. In particular it seems like https://github.com/containers/storage/issues/1928 is somewhat close to a blocker for wide scale deployment.