moby
Compression parameters ignored
The compression algorithm parameters are completely ignored when using buildx to build and push an image. This is despite following the example command in the compression documentation. The size and media type in the manifest remain the same regardless of the compression algorithm I choose.
It appears the compression parameters are not being read at all because if I provide invalid values (e.g. a non-numeric compression-level) I do not get an error.
I have tested on both WSL (Ubuntu) and Windows. The below sequence of commands demonstrates the problem.
BuildKit version
PS C:\Users\usrname> docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default docker
\_ default \_ default running v0.15.2 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux* docker
\_ desktop-linux \_ desktop-linux running v0.15.2 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
Dockerfile
PS C:\Users\usrname> cat Dockerfile
FROM alpine
ARG ID
RUN cp /bin/busybox /home
Build and push image compressed with gzip
PS C:\Users\usrname> docker buildx build --build-arg ID=gzip --output type=image,name=usrname/dummy:gzip,push=true,compression=gzip .
[+] Building 23.7s (7/7) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 85B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/2] FROM docker.io/library/alpine:latest 0.0s
=> [2/2] RUN cp /bin/busybox /home 0.3s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:2d1df90518de65637f7798eed87f30ad75a547e0e6d30d6acc5b5547731afc3e 0.0s
=> => naming to docker.io/usrname/dummy:gzip 0.0s
=> pushing usrname/dummy:gzip with docker 17.3s
=> => pushing layer e2696caccd8d 7.9s
=> => pushing layer 94e5f06ff8e3 15.6s
Build and push image compressed with zstd
PS C:\Users\usrname> docker buildx build --build-arg ID=zstd --output type=image,name=usrname/dummy:zstd,push=true,compression=zstd .
[+] Building 23.0s (7/7) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 85B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/2] FROM docker.io/library/alpine:latest 0.0s
=> [2/2] RUN cp /bin/busybox /home 0.2s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:e211987a585928034e89c408c7cdc6f15a77a2e6b45b4616cec10f0e5dfab665 0.0s
=> => naming to docker.io/usrname/dummy:zstd 0.0s
=> pushing usrname/dummy:zstd with docker 17.0s
=> => pushing layer 7b5d89f5975c 7.7s
=> => pushing layer 94e5f06ff8e3 15.8s
Build and push uncompressed image
PS C:\Users\usrname> docker buildx build --build-arg ID=uncompressed --output type=image,name=usrname/dummy:uncompressed,push=true,compression=uncompressed .
[+] Building 23.4s (7/7) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 85B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/2] FROM docker.io/library/alpine:latest 0.0s
=> [2/2] RUN cp /bin/busybox /home 0.2s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d8e8715782f9d17423d267590c2ac74dda8743621a5910a067ffdf3073fde711 0.0s
=> => naming to docker.io/usrname/dummy:uncompressed 0.0s
=> pushing usrname/dummy:uncompressed with docker 17.1s
=> => pushing layer b2ffd68f6199 7.9s
=> => pushing layer 94e5f06ff8e3 15.8s
Manifest for gzip image
PS C:\Users\usrname> docker manifest inspect usrname/dummy:gzip
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 939,
"digest": "sha256:2d1df90518de65637f7798eed87f30ad75a547e0e6d30d6acc5b5547731afc3e"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3623844,
"digest": "sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 513591,
"digest": "sha256:958ef0a9c069b82a13223ae4be9b6bfaeebdfa4070dc7045135b3c1db199da1a"
}
]
}
Manifest for zstd image
PS C:\Users\usrname> docker manifest inspect usrname/dummy:zstd
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 939,
"digest": "sha256:e211987a585928034e89c408c7cdc6f15a77a2e6b45b4616cec10f0e5dfab665"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3623844,
"digest": "sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 513590,
"digest": "sha256:d3792e8213e6ac1b7362242f6d4ca227bafa11bb3ad1836d86e483c46fb7d137"
}
]
}
Manifest for uncompressed image
PS C:\Users\usrname> docker manifest inspect usrname/dummy:uncompressed
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 955,
"digest": "sha256:d8e8715782f9d17423d267590c2ac74dda8743621a5910a067ffdf3073fde711"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3623844,
"digest": "sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 513589,
"digest": "sha256:4b50faf70afcaad978d835f74ac88d589952c2e9e2e9454ed84db9c3c2ca7b65"
}
]
}
Setting compression options requires container builder or docker instance with containerd storage. https://docs.docker.com/desktop/containerd/ . Pre-containerd the layer blobs are not created during build but when running docker push.
If the build references a blob that already exists, eg. from a base image or via existing build cache then you need to set force-compression if you wish to reencode the existing blobs.
Setting compression options requires container builder or docker instance with containerd storage. https://docs.docker.com/desktop/containerd/ . Pre-containerd the layer blobs are not created during build but when running docker push.
This is not what's written in the documentation. See https://docs.docker.com/build/exporters/#compression
If you statement is true, why is there no error message?
Moreover, according to the documentation, compression can only be set when building images. There is no option to set compression options when pushing layers to a registry.
All docker instances nowadays run on containerd.
Moreover, according to the documentation, compression can only be set when building images. There is no option to set compression options when pushing layers to a registry.
Idk what you mean by pushing layers here. Layers are pushed as part of the image.
All docker instances nowadays run on containerd.
Containerd storage is not enabled by default on dockerd. https://docs.docker.com/engine/storage/containerd/
I see multiple action items here:
- document that selecting the compression algo and level depends on an experimental feature
- logging an error if a feature is requested that is unsupported by the current feature set
WDYT?
