opencontainers
performance: what can dist-spec do to improve downloads of large images/layers?
Things we already allow/do:
- Parallel download of layers
Things we can likely improve:
- For large layers, range-based downloads - download sections of a large file using Range Header and stitch them back together?
Things known to the community:
- https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md ^ fuse based filesystem solutions - copy individual things when referenced
For streaming/lazy loading:
- GKE image streaming
- https://github.com/awslabs/soci-snapshotter
Multipart layer downloads with range requests:
- https://github.com/containerd/containerd/pull/10177
- https://github.com/awslabs/amazon-ecr-containerd-resolver#parallel-downloads
Direct mounts of compressed tars (saves on extraction time):
- https://github.com/containerd/containerd/pull/9362#issuecomment-1809829145
- https://github.com/containers/composefs
What changes are needed in distribution-spec to support this? Is a pointer to the HTTP specs documenting range requests enough?
What changes are needed in distribution-spec to support this? Is a pointer to the HTTP specs documenting range requests enough?
IMO, what server/client side optimizations can be enabled by dist-spec changes? demonstrably?
Just like conformance, we should write benchmark code for this.
Just like conformance, we should write benchmark code for this.
Is that an OCI requirement, or something implementations should be doing?
Just like conformance, we should write benchmark code for this.
Is that an OCI requirement, or something implementations should be doing?
Not an OCI requirement - our conformance suite should already ensure conformant registries can handle range-based blob pulls.
https://github.com/opencontainers/distribution-spec/pull/537#issuecomment-2155821558 ^ will be interesting to see data from various registries. If viable, clients should move to this model advised by blob size.