buildkit
buildkit copied to clipboard
moby
Reduce verbosity of image pull steps
Relates to https://github.com/docker/compose/issues/8800
The output of docker build with BuildKit is quite verbose if an image is pulled as part of the build;
echo 'FROM postgres' | docker build -
[+] Building 13.4s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 56B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/postgres:latest 2.7s
=> [auth] library/postgres:pull token for registry-1.docker.io 0.0s
=> [1/1] FROM docker.io/library/postgres@sha256:f76241d07218561e3d1a334eae6a5bf63c70b49f35ffecb7f020448e30e37390 10.6s
=> => resolve docker.io/library/postgres@sha256:f76241d07218561e3d1a334eae6a5bf63c70b49f35ffecb7f020448e30e37390 0.0s
=> => sha256:e5ae68f740265288a4888db98d2999a638fdcb6d725f427678814538d253aa4d 31.37MB / 31.37MB 1.4s
=> => sha256:f76241d07218561e3d1a334eae6a5bf63c70b49f35ffecb7f020448e30e37390 1.86kB / 1.86kB 0.0s
=> => sha256:e94a3bb612246f1f672a0d11fbd16415e2f95d308b37d38deaa8c2bd3c0116d8 10.23kB / 10.23kB 0.0s
=> => sha256:7527d03e2f7758fcbc420254a6a9ae51b970e70fec727269376356568f42e9bc 1.80kB / 1.80kB 0.4s
=> => sha256:7b8fcc7e1ad054463615f8e9ada48d0c011c51bc03317a709c4cfc23a3af52c7 4.41MB / 4.41MB 0.7s
=> => sha256:fb0630c9679aeef051ca89dddf5919c19be96109ae0648116900be09eb79545e 3.04kB / 3.04kB 0.0s
=> => sha256:80e55689f4d0cdd390957c8e1135b143ca3afb1486a1ca5a9fc01429d483b48d 1.42MB / 1.42MB 0.9s
=> => sha256:8a79eb6d69c9c99a8e75aa15011c9e57d0af9f3822905c12ec38e68d9d5c5cb9 8.05MB / 8.05MB 1.5s
=> => sha256:397705f2d09375da10b9c3cbfe61556a95d9673f6e016382f20bfed7284e85db 441.55kB / 441.55kB 1.2s
=> => sha256:de36ec4eb0a50925495a0bbc72e83cab5bd5d8ecf490c913f13412b2786fc25e 149B / 149B 1.5s
=> => sha256:08d878a022c1a8a3333ae9a3de8170431bde517831abbca34f768501e5cfda51 3.05kB / 3.05kB 1.6s
=> => sha256:7677029670ff4fe3625939c77c161e208cdc1e9c21ff9095d23160380ff492e5 91.23MB / 91.23MB 4.6s
=> => extracting sha256:e5ae68f740265288a4888db98d2999a638fdcb6d725f427678814538d253aa4d 2.7s
=> => sha256:1d24b3d9557e5acd875742dd0d101e562a5f2ca32ed7cc3351d3b4d8bb8bed7a 9.54kB / 9.54kB 1.7s
=> => sha256:e085b018338cc45b376fb135133a09b9736b5ea35be6d9925dc8fbde17e7e98b 129B / 129B 1.8s
=> => sha256:063b09ff12e95630f4462889f6f1fb572de84bbde104ea7f469d6497799a9736 201B / 201B 1.9s
=> => sha256:a39fee215a44ffe3b744f8a79378afe57a00606a5f28856ddbd0096000c0d95d 4.72kB / 4.72kB 2.0s
=> => extracting sha256:7b8fcc7e1ad054463615f8e9ada48d0c011c51bc03317a709c4cfc23a3af52c7 0.2s
=> => extracting sha256:7527d03e2f7758fcbc420254a6a9ae51b970e70fec727269376356568f42e9bc 0.1s
=> => extracting sha256:80e55689f4d0cdd390957c8e1135b143ca3afb1486a1ca5a9fc01429d483b48d 0.1s
=> => extracting sha256:8a79eb6d69c9c99a8e75aa15011c9e57d0af9f3822905c12ec38e68d9d5c5cb9 0.4s
=> => extracting sha256:397705f2d09375da10b9c3cbfe61556a95d9673f6e016382f20bfed7284e85db 0.1s
=> => extracting sha256:de36ec4eb0a50925495a0bbc72e83cab5bd5d8ecf490c913f13412b2786fc25e 0.0s
=> => extracting sha256:08d878a022c1a8a3333ae9a3de8170431bde517831abbca34f768501e5cfda51 0.0s
=> => extracting sha256:7677029670ff4fe3625939c77c161e208cdc1e9c21ff9095d23160380ff492e5 3.7s
=> => extracting sha256:1d24b3d9557e5acd875742dd0d101e562a5f2ca32ed7cc3351d3b4d8bb8bed7a 0.0s
=> => extracting sha256:e085b018338cc45b376fb135133a09b9736b5ea35be6d9925dc8fbde17e7e98b 0.0s
=> => extracting sha256:063b09ff12e95630f4462889f6f1fb572de84bbde104ea7f469d6497799a9736 0.0s
=> => extracting sha256:a39fee215a44ffe3b744f8a79378afe57a00606a5f28856ddbd0096000c0d95d 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d2474174ccb2b79d24c4a02130c602d0800484df71818a93ae2aff9adc651663 0.0s
While pull progress of individual layers may be useful while it's pulling, perhaps it would be possible to;
- limit the number of lines shown (similar to how
RUNoutput uses a limited number of lines that scroll) - collapse the pull details after the pull completed (after which the information may no longer be relevant if successful)
Perhaps when collapsing, a summary should be shown;
- amount of data pulled
- time taken
- image name/tag and digest
- perhaps platform (
os/arch) as well
let me /cc @jedevc as well (for more context, see https://github.com/docker/compose/issues/8800)
With docker-container driver (buildkit 0.10.3):
echo 'FROM postgres' | docker buildx build -
[+] Building 1.8s (4/4) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 51B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/postgres:latest 1.1s
=> [1/1] FROM docker.io/library/postgres@sha256:9ceb24f8c5f15c053d973a3610866f473690875dc13eb3282b45302189321040 0.6s
=> => resolve docker.io/library/postgres@sha256:9ceb24f8c5f15c053d973a3610866f473690875dc13eb3282b45302189321040
With docker driver (docker 20.10.17 ~> buildkit 0.8.2)
echo 'FROM postgres' | docker buildx build -
[+] Building 8.7s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 51B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/postgres:latest 1.5s
=> [auth] library/postgres:pull token for registry-1.docker.io 0.0s
=> [1/1] FROM docker.io/library/postgres@sha256:9ceb24f8c5f15c053d973a3610866f473690875dc13eb3282b45302189321040 7.0s
=> => resolve docker.io/library/postgres@sha256:9ceb24f8c5f15c053d973a3610866f473690875dc13eb3282b45302189321040 0.0s
=> => sha256:e09e90144645e02137d087f0dc059f4d2e3c6356ef8f9e40eeb15d1c901dbc73 10.51kB / 10.51kB 0.0s
=> => sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB 1.0s
=> => sha256:9ceb24f8c5f15c053d973a3610866f473690875dc13eb3282b45302189321040 1.86kB / 1.86kB 0.0s
=> => sha256:66c520df917d3715e4642387ce8786797c184e9f415a1aeab04e5dd070022702 4.41MB / 4.41MB 0.4s
=> => sha256:5b124e6748c9fe30c3e6483ac420fe044a00ad7d38b51783104a8276360ba8ad 1.80kB / 1.80kB 0.2s
=> => sha256:cf3b0cf1dde2a82542e4b9de7f3ad058fdc819dea6499007035b838542b0bd5e 3.04kB / 3.04kB 0.0s
=> => sha256:1a06bb042d01cf46ad9ea6c04b8a55c2a6b880c64cbfb887e2d4ae1f42765bfb 1.42MB / 1.42MB 0.5s
=> => sha256:e3849c675ec5f366fccdecb7e849db6ab96b31f0d19104b8aaac2e51fe5ca1a2 8.05MB / 8.05MB 0.8s
=> => sha256:d3b2eaf1435b467f692b57d4474b688b9597a8789a1593b6f3a82c386b21d5c8 1.26MB / 1.26MB 0.7s
=> => sha256:074399829dc9a56b8d422dc00806dbff19fef84dfaaebd8a0ee891057e34111f 149B / 149B 0.8s
=> => sha256:02f61377c93d363a14ea4d8e7d95d272cb5e7339745fe44ff6cc1035e800f03b 91.28MB / 91.28MB 3.9s
=> => sha256:6feb085525d8c7a128d17fe5f690efaae1aa9f6b3cfe722e409f7e2d775a1d9f 3.20kB / 3.20kB 1.0s
=> => sha256:d639033f0429d74740a2edfae54e72e95aae39be6443bdb59ea9d7bd88428c47 9.54kB / 9.54kB 1.1s
=> => extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 0.9s
=> => sha256:1a040507c35bc1a8980e5411bb975d98c9fc41964b16f40096a3755e6607e5cc 129B / 129B 1.2s
=> => sha256:11209ef4fb4edbac3d2b1990c7e46717262bf30e759be9a72aa9e8b55601efc1 4.70kB / 4.70kB 1.3s
=> => sha256:aa2ec30a462da8c5629703ece571f574c6c90c4d98a94b3baef0486367b4eeb1 201B / 201B 1.3s
=> => extracting sha256:66c520df917d3715e4642387ce8786797c184e9f415a1aeab04e5dd070022702 0.1s
=> => extracting sha256:5b124e6748c9fe30c3e6483ac420fe044a00ad7d38b51783104a8276360ba8ad 0.0s
=> => extracting sha256:1a06bb042d01cf46ad9ea6c04b8a55c2a6b880c64cbfb887e2d4ae1f42765bfb 0.1s
=> => extracting sha256:e3849c675ec5f366fccdecb7e849db6ab96b31f0d19104b8aaac2e51fe5ca1a2 0.2s
=> => extracting sha256:d3b2eaf1435b467f692b57d4474b688b9597a8789a1593b6f3a82c386b21d5c8 0.1s
=> => extracting sha256:074399829dc9a56b8d422dc00806dbff19fef84dfaaebd8a0ee891057e34111f 0.0s
=> => extracting sha256:6feb085525d8c7a128d17fe5f690efaae1aa9f6b3cfe722e409f7e2d775a1d9f 0.0s
=> => extracting sha256:02f61377c93d363a14ea4d8e7d95d272cb5e7339745fe44ff6cc1035e800f03b 2.0s
=> => extracting sha256:d639033f0429d74740a2edfae54e72e95aae39be6443bdb59ea9d7bd88428c47 0.0s
=> => extracting sha256:1a040507c35bc1a8980e5411bb975d98c9fc41964b16f40096a3755e6607e5cc 0.0s
=> => extracting sha256:aa2ec30a462da8c5629703ece571f574c6c90c4d98a94b3baef0486367b4eeb1 0.0s
=> => extracting sha256:11209ef4fb4edbac3d2b1990c7e46717262bf30e759be9a72aa9e8b55601efc1 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:892c691fdb0ad3e990e41d27853a502b1655c104a4c53475ff2bc67cb2e371f9
@thaJeztah So I would say it will be fixed with next 22.06
@crazy-max I believe that is just lazy loading in your output there. You are not exporting image so layers are not pulled.
A possibly quicker optimization for this that doesn't involve rethinking the whole progressbar would be to collapse all these extracting rows into the same line. Extraction happens serially, so there isn't really any value in showing them separately, could be just one line that says "extracting image layers".
I believe that is just lazy loading in your output there
Good catch; yes probably changing it to FROM postgres\nRUN echo foo would show a pull happening
could be just one line that says "extracting image layers".
Yup, sounds like at least in the right direction (perhaps 1/x)?
Extraction happens serially
I think the downloading (<digest> 4.41MB / 4.41MB ) still happens in parallel?
I think the downloading (
4.41MB / 4.41MB ) still happens in parallel?
Yes
LOL, I was about to open a ticket for this, and then realized I already had one for this. Just tested output on Docker 23.0.1 (BuildKit v0.10);
# syntax=docker/dockerfile:1
FROM alpine AS one
RUN echo hello > /foo.txt
FROM golang AS two
RUN ls -la / && cd /usr && ls -la && echo bar > /bar.txt
FROM scratch AS final
COPY --from=one /foo.txt .
COPY --from=two /bar.txt .
Which shows:
[+] Building 21.7s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 268B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.9s
=> docker-image://docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14 0.6s
=> => resolve docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14 0.0s
=> => sha256:a47ff7046597eea0123ea02817165350e3680f75000dc5d69c9a310258e1bedd 11.55MB / 11.55MB 0.2s
=> => sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14 8.40kB / 8.40kB 0.0s
=> => sha256:966d40f9ba8366e74c2fa353fc0bc7bbc167d2a0f3ad2420db8b9e633049462d 482B / 482B 0.0s
=> => sha256:dbdd11720762ad504260c66161c964e59eba06b95a7aa64a68634b598a830a91 2.90kB / 2.90kB 0.0s
=> => extracting sha256:a47ff7046597eea0123ea02817165350e3680f75000dc5d69c9a310258e1bedd 0.3s
=> [internal] load metadata for docker.io/library/golang:latest 0.9s
=> [internal] load metadata for docker.io/library/alpine:latest 0.9s
=> [one 1/2] FROM docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a 2.0s
=> => resolve docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a 0.1s
=> => sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a 1.64kB / 1.64kB 0.0s
=> => sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501 528B / 528B 0.0s
=> => sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d 1.47kB / 1.47kB 0.0s
=> => sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB 0.2s
=> => extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 1.4s
=> [two 1/2] FROM docker.io/library/golang@sha256:9911e9376e45de1ba865b16d878c0f7d063391fe4b7cedf39869874126741e62 17.9s
=> => resolve docker.io/library/golang@sha256:9911e9376e45de1ba865b16d878c0f7d063391fe4b7cedf39869874126741e62 0.1s
=> => sha256:1e4aec178e0864db93a6f97a20bde3445871a4562c1801185eca1238d3a0e80d 55.05MB / 55.05MB 0.7s
=> => sha256:6c1024729feeb2893dad43684fe7679c4d866c3640dfc3912bbd93c5a51f32d2 5.17MB / 5.17MB 0.3s
=> => sha256:9911e9376e45de1ba865b16d878c0f7d063391fe4b7cedf39869874126741e62 2.36kB / 2.36kB 0.0s
=> => sha256:36eee7cb5eb815b82ce42611d6f5f51e4ebff23d6a4fedc4f2d1d5e3ee213326 7.21kB / 7.21kB 0.0s
=> => sha256:d19ee8512191c8b8e967246d4a7d0de4f4133a30bd9ce982e9b70a0c596dbf18 1.79kB / 1.79kB 0.0s
=> => sha256:c3aa11fbc85a2a9660c98cfb4d0a2db8cde983ce3c87565c356cfdf1ddf2654c 10.88MB / 10.88MB 0.5s
=> => sha256:aa54add66b3a47555c8b761f60b15f818236cc928109a30032111efc98c6fcd4 54.59MB / 54.59MB 1.2s
=> => sha256:bb80f942c8f48f157732f2928b6d37b63b9a9989bbcd6cd0f21a18ce292fc100 85.99MB / 85.99MB 1.8s
=> => sha256:b79d6fbafb4b27d2ac2a0aca3484b5e7c242dadab2c201947ddd249014b89b72 99.89MB / 99.89MB 2.4s
=> => extracting sha256:1e4aec178e0864db93a6f97a20bde3445871a4562c1801185eca1238d3a0e80d 4.0s
=> => sha256:673b1967f3b3d1eb54fdb166a8f9adfacdc9ea6519df76da829ef49faae95177 155B / 155B 1.5s
=> => extracting sha256:6c1024729feeb2893dad43684fe7679c4d866c3640dfc3912bbd93c5a51f32d2 0.3s
=> => extracting sha256:c3aa11fbc85a2a9660c98cfb4d0a2db8cde983ce3c87565c356cfdf1ddf2654c 0.3s
=> => extracting sha256:aa54add66b3a47555c8b761f60b15f818236cc928109a30032111efc98c6fcd4 2.9s
=> => extracting sha256:bb80f942c8f48f157732f2928b6d37b63b9a9989bbcd6cd0f21a18ce292fc100 3.3s
=> => extracting sha256:b79d6fbafb4b27d2ac2a0aca3484b5e7c242dadab2c201947ddd249014b89b72 5.0s
=> => extracting sha256:673b1967f3b3d1eb54fdb166a8f9adfacdc9ea6519df76da829ef49faae95177 0.0s
=> [one 2/2] RUN echo hello > /foo.txt 0.6s
=> [final 1/2] COPY --from=one /foo.txt . 0.0s
=> [two 2/2] RUN ls -la / && cd /usr && ls -la && echo bar > /bar.txt 0.5s
=> [final 2/2] COPY --from=two /bar.txt . 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:0a3ff7b4e669d0666fec7824734ff5fb6a8e053ae915766f9233255d3a7f5e03 0.0s
=> => naming to docker.io/library/foo 0.0s
I still think we could consider hiding pull progress after it completed, like we do for RUN steps (like the example above which does some ls -la as part of the step, but after that completed successfully, we hide the output), which would reduce the final output to something like;
[+] Building 21.7s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 268B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.9s
=> docker-image://docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14 0.6s
=> [internal] load metadata for docker.io/library/golang:latest 0.9s
=> [internal] load metadata for docker.io/library/alpine:latest 0.9s
=> [one 1/2] FROM docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a 2.0s
=> [two 1/2] FROM docker.io/library/golang@sha256:9911e9376e45de1ba865b16d878c0f7d063391fe4b7cedf39869874126741e62 17.9s
=> [one 2/2] RUN echo hello > /foo.txt 0.6s
=> [final 1/2] COPY --from=one /foo.txt . 0.0s
=> [two 2/2] RUN ls -la / && cd /usr && ls -la && echo bar > /bar.txt 0.5s
=> [final 2/2] COPY --from=two /bar.txt . 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:0a3ff7b4e669d0666fec7824734ff5fb6a8e053ae915766f9233255d3a7f5e03 0.0s
=> => naming to docker.io/library/foo 0.0s
Perhaps also consider swapping "pull" output for "extracting" output (like docker pull does (I think?)), i.e., swap this output;
=> => sha256:1e4aec178e0864db93a6f97a20bde3445871a4562c1801185eca1238d3a0e80d 55.05MB / 55.05MB 0.7s
For the "extracting" output;
=> => extracting sha256:1e4aec178e0864db93a6f97a20bde3445871a4562c1801185eca1238d3a0e80d 4.0s
A summary (platform? total size?) could still be useful information I guess. We do show this for Build Context, Dockerfile and even .dockerignore file (fun fact: there's actually no .dockerignore in the example above, so that's confusing - I'll open a ticket for that).