buildkit
buildkit copied to clipboard
Published
20 hours ago •
moby
moby
Image Layers Going Missing
Ubuntu Jammy just did an unattended upgrade:
Start-Date: 2023-02-03 08:54:33
Commandline: aptdaemon role='role-commit-packages' sender=':1.38787'
Install: docker-compose-plugin:amd64 (2.15.1-1~ubuntu.22.04~jammy, automatic), libllvm15:amd64 (1:15.0.6-3~ubuntu0.22.04.2, automatic), docker-buildx-plugin:amd64 (0.10.2-1~ubuntu.22.04~jammy, automatic)
Upgrade: containerd.io:amd64 (1.6.15-1, 1.6.16-1), docker-ce-cli:amd64 (5:20.10.23~3-0~ubuntu-jammy, 5:23.0.0-1~ubuntu.22.04~jammy), libglx-mesa0:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), gnome-control-center-faces:amd64 (1:41.7-0ubuntu0.22.04.5, 1:41.7-0ubuntu0.22.04.6), kbd:amd64 (2.3.0-3ubuntu4, 2.3.0-3ubuntu4.22.04), libgbm1:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), grub-pc-bin:amd64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), libdrm-nouveau2:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), libinput10:amd64 (1.20.0-1ubuntu0.1, 1.20.0-1ubuntu0.2), libxatracker2:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), libegl1-mesa:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), mesa-va-drivers:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), libgl1-mesa-dri:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), docker-ce:amd64 (5:20.10.23~3-0~ubuntu-jammy, 5:23.0.0-1~ubuntu.22.04~jammy), libgl1-mesa-glx:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), libdrm-common:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), docker-ce-rootless-extras:amd64 (5:20.10.23~3-0~ubuntu-jammy, 5:23.0.0-1~ubuntu.22.04~jammy), mesa-vulkan-drivers:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), gnome-control-center-data:amd64 (1:41.7-0ubuntu0.22.04.5, 1:41.7-0ubuntu0.22.04.6), grub2-common:amd64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), libglapi-mesa:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), grub-common:amd64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), libdrm-amdgpu1:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), libdrm-radeon1:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), libdrm2:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), gnome-control-center:amd64 (1:41.7-0ubuntu0.22.04.5, 1:41.7-0ubuntu0.22.04.6), libegl-mesa0:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), mesa-vdpau-drivers:amd64 (22.0.5-0ubuntu0.3, 22.2.5-0ubuntu0.1~22.04.1), grub-pc:amd64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), libdrm-intel1:amd64 (2.4.110-1ubuntu1, 2.4.113-2~ubuntu0.22.04.1), libinput-bin:amd64 (1.20.0-1ubuntu0.1, 1.20.0-1ubuntu0.2)
End-Date: 2023-02-03 08:55:15
And that included a bunch of docker goodies. However my builds suddenly and inexplicably stopped functioning. I've managed to do a quick PoC:
[Fri 3 Feb 11:32:15 GMT 2023] simon@symphony ~/temp ls
Dockerfile file
[Fri 3 Feb 11:32:15 GMT 2023] simon@symphony ~/temp cat Dockerfile
FROM ubuntu:jammy as base
RUN apt update \
&& apt -y install ca-certificates
FROM scratch
COPY --from=base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY file /etc/passwd
Now without the "file" line, everything is dandy, with it present as above:
[Fri 3 Feb 11:32:17 GMT 2023] simon@symphony ~/temp docker build . -t test/test:0.0.0
[+] Building 0.1s (9/9) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 240B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:jammy 0.0s
=> [base 1/2] FROM docker.io/library/ubuntu:jammy 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 23B 0.0s
=> CACHED [base 2/2] RUN apt update && apt -y install ca-certificates 0.0s
=> CACHED [stage-1 1/2] COPY --from=base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-cer 0.0s
=> CACHED [stage-1 2/2] COPY file /etc/passwd 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1b7f4efd79b48f31afd34293c21fe5d5d1ab711e7feb15ed4437879fecfedf85 0.0s
=> => naming to docker.io/test/test:0.0.0 0.0s
[Fri 3 Feb 11:32:42 GMT 2023] simon@symphony ~/temp docker save test/test:0.0.0 > image.tar
[Fri 3 Feb 11:33:11 GMT 2023] simon@symphony ~/temp tar xf image.tar
[Fri 3 Feb 11:33:15 GMT 2023] simon@symphony ~/temp ls
1b7f4efd79b48f31afd34293c21fe5d5d1ab711e7feb15ed4437879fecfedf85.json file repositories
d971a20ba5fff0590a04129e0f198b9ed6a76eb56598f557fa7b27b95eeed662 image.tar
Dockerfile manifest.json
[Fri 3 Feb 11:33:17 GMT 2023] simon@symphony ~/temp tar tf d971a20ba5fff0590a04129e0f198b9ed6a76eb56598f557fa7b27b95eeed662/layer.tar
etc/
etc/passwd
[Fri 3 Feb 11:33:26 GMT 2023] simon@symphony ~/temp docker image history test/test:0.0.0
IMAGE CREATED CREATED BY SIZE COMMENT
1b7f4efd79b4 5 minutes ago COPY file /etc/passwd # buildkit 0B buildkit.dockerfile.v0
<missing> 5 minutes ago COPY /etc/ssl/certs/ca-certificates.crt /etc… 0B buildkit.dockerfile.v0
So the question is where the hell have my CA certificates gone??
Note: as a work around I can install the certs into / but evidently not a sub directory.
What changed is in the dpkg logs above. However to shortcut it, I do see docker-buildx-plugin:amd64 (0.10.2-1~ubuntu.22.04~jammy, automatic) which is the same as the 0.10 discussed in the attached PR, and does sound remarkably similar.
Here's a full dump of dpkg
dpkg -l | grep docker
ii docker-buildx-plugin 0.10.2-1~ubuntu.22.04~jammy amd64 Docker Buildx cli plugin.
ii docker-ce 5:23.0.0-1~ubuntu.22.04~jammy amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:23.0.0-1~ubuntu.22.04~jammy amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:23.0.0-1~ubuntu.22.04~jammy amd64 Rootless support for Docker.
ii docker-compose-plugin 2.15.1-1~ubuntu.22.04~jammy amd64 Docker Compose (V2) plugin for the Docker CLI.
ii docker-scan-plugin 0.23.0~ubuntu-jammy amd64 Docker scan cli plugin.
rc docker.io 20.10.12-0ubuntu4 amd64 Linux container runtime
I'm guessing from the evidence, Canonical finally decided to package buildx, overwrite my manual install and break it 😸 Let me check if there's a related issue on their side...
