buildkit
buildkit copied to clipboard
Published
20 hours ago •
moby
moby
Cache miss when using the image that has been built from the another cache
Hello,
I'm using Docker version 20.10.17, build 100c701 on WSL2 with Ubuntu (5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux).
I encountered a strange behavior that the cache miss occurs when using the image that has been built from the another cache.
Steps to reproduce:
- Run a Docker registry for storing images.
- Build a fresh Docker image (a first one) and push it to the registry.
- Change Docker context to cause a Docker cache miss in the only one layer.
- Build a new Docker image (a second one) using cache from the first one and push it to the registry. Only one (the latest) layer has to be rebuilt.
- Change Docker context to cause a Docker cache miss in the only one layer.
- Build a new Docker image (a third one) using cache from the second one and push it to the registry. Only one (the latest) layer has to be rebuilt. But it rebuilds the whole image - all layers.
To simplify the issue explanation I've created a small repository with the reproduce.sh script - https://github.com/AlexPykavy/docker-cache-miss-issue.
Checkout the output:
+ echo '************************* Starting Docker registry *************************'
************************* Starting Docker registry *************************
++ pwd
+ registry_mount=/mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry
+ docker run -d -p 5000:5000 -v /mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry:/var/lib/registry --restart=always --name registry registry:2
7bf2139bb35502398292069fda069b4c76c0705ac2263adb5922ff140f3722cf
+ echo '************************* Building initial image *************************'
************************* Building initial image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:3
deleted: sha256:33b4ff8cd367e2483ae4dc1853858d34050b95d52ee7a6cdaabb20b0c76caab3
Deleted build cache objects:
tgzhc2zqjf2wfnqb7tzhgz85t
pedx98is8tj34g129qgpz98ye
q9ub8btj54my6iq3k1pkyk49m
tv961itpyh47y12k0bhkrku2p
bxbfz16zl8vdh0hgludpzvrvk
to93frbzk2em5axqi3b07ew6n
vmyksjeas1dsr18o549egwokv
9rjdjsmbyn79cneaj4rs3ltxg
eciei23i0azt4qkisuv3qgqs3
v77r0jl3o37dthnly80vmrarp
zkdr46ir26lba3vnarjeomgba
n0i6d7kkgdfo0cjkv84sdw3m6
Total reclaimed space: 12MB
+ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
registry 2 d1fe2eaf6101 2 weeks ago 24.1MB
+ docker build -t localhost:5000/cache-experiments:1 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 95.6s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 229B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye 88.1s
=> [internal] load build context 0.0s
=> => transferring context: 368B 0.0s
=> [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 3.9s
=> => resolve docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 0.0s
=> => sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 1.08MB / 1.08MB 0.7s
=> => sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 11.77MB / 11.77MB 1.1s
=> => sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 1.86kB / 1.86kB 0.0s
=> => sha256:bdf0079de4094afdb26b94d9f89b716499436282c972461d945a87899c015c23 1.37kB / 1.37kB 0.0s
=> => sha256:5b7d04484c737c4a2af7f33a85470dab369e922a570411f40ebd72d35167bc49 7.49kB / 7.49kB 0.0s
=> => sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB 1.5s
=> => sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 233B / 233B 0.9s
=> => sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 3.18MB / 3.18MB 1.4s
=> => extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 1.2s
=> => extracting sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 0.1s
=> => extracting sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 0.4s
=> => extracting sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 0.0s
=> => extracting sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 0.2s
=> [2/5] WORKDIR /app 0.2s
=> [3/5] COPY requirements.txt ./ 0.0s
=> [4/5] RUN pip install -r requirements.txt 3.1s
=> [5/5] COPY src ./ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:e5871e1a72635ddcd169ce3930e91647adedc36cd9c561e308b548fd0a89b8d4 0.0s
=> => naming to localhost:5000/cache-experiments:1 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ docker push localhost:5000/cache-experiments:1
The push refers to repository [localhost:5000/cache-experiments]
6bc1a8c56437: Pushed
f30a8f1d517a: Pushed
10ed798bb875: Pushed
e78c457618a4: Pushed
b4573ffeefec: Pushed
5ebe2abcebb0: Pushed
7287cceb8eb6: Pushed
e7a38ce55a0d: Pushed
92a4e8a3140f: Pushed
1: digest: sha256:9250525ab23e71a3f8ba223555624a54374876f5d60cff98c0ce66f76e4f92a5 size: 2202
+ echo '************************* Changing the Docker context *************************'
************************* Changing the Docker context *************************
+ echo import os
+ echo '************************* Building the second image *************************'
************************* Building the second image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:1
untagged: localhost:5000/cache-experiments@sha256:9250525ab23e71a3f8ba223555624a54374876f5d60cff98c0ce66f76e4f92a5
deleted: sha256:e5871e1a72635ddcd169ce3930e91647adedc36cd9c561e308b548fd0a89b8d4
Deleted build cache objects:
3ik09luezfct8zraputleixyj
ssv1ygz4aga5222a66h9fhc8z
komliwdehakp13kawekurkdny
i3epw8wwi4h9abkdf5p06rd3a
rvg3z6021s9baf4abfn98pxjf
mps7fpoaodb6ao3ddtoea50de
z34qhtdiklae00o9nlu6czxt0
wmq6bk2e6knkaopy0c094kg7z
kuqsy5xpwd9z037r58fy0kp5b
t812pl4jyzhx1mt8nox4h244w
oz1c1sojyg0cftpvz0606an9r
jp29eb5sfqzq5uvd09bqly6a0
Total reclaimed space: 12MB
+ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
registry 2 d1fe2eaf6101 2 weeks ago 24.1MB
+ docker build -t localhost:5000/cache-experiments:2 --cache-from localhost:5000/cache-experiments:1 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 47.3s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 229B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye 43.7s
=> importing cache manifest from localhost:5000/cache-experiments:1 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 418B 0.0s
=> [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 0.0s
=> CACHED [2/5] WORKDIR /app 0.0s
=> CACHED [3/5] COPY requirements.txt ./ 0.0s
=> CACHED [4/5] RUN pip install -r requirements.txt 3.1s
=> => pulling sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 0.0s
=> => pulling sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 0.1s
=> => pulling sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 0.2s
=> => pulling sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 0.0s
=> => pulling sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 0.1s
=> => pulling sha256:5f71eb90fb301cc7600aa81f89eabc0a135cfe8dfede2488f93abc3f74be80a2 0.0s
=> => pulling sha256:46ccdbc34fc33a74ae69bbaed8d3a1b339db81a5c231ef5646d414a424bf8fb2 0.0s
=> => pulling sha256:612fc9be3d770781610df94fe2336cc63851fe9ea7ca2b919299b331b01565ae 0.1s
=> [5/5] COPY src ./ 0.2s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1ed2f6427ff9596dd94065719b5d9bc7c5be85c703b4ef0c41ecc30c2fd6c1d6 0.0s
=> => naming to localhost:5000/cache-experiments:2 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ docker push localhost:5000/cache-experiments:2
The push refers to repository [localhost:5000/cache-experiments]
0730abb3e72a: Pushed
f30a8f1d517a: Layer already exists
10ed798bb875: Layer already exists
e78c457618a4: Layer already exists
b4573ffeefec: Layer already exists
5ebe2abcebb0: Layer already exists
7287cceb8eb6: Layer already exists
e7a38ce55a0d: Layer already exists
92a4e8a3140f: Layer already exists
2: digest: sha256:ba6ddefecbc8cbc931dfe013dde59c111f9332837592a47f257105cb7f24df82 size: 2202
+ echo '************************* Changing the Docker context *************************'
************************* Changing the Docker context *************************
+ rm -f src/temp.py
+ echo '************************* Building the third image *************************'
************************* Building the third image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:2
untagged: localhost:5000/cache-experiments@sha256:ba6ddefecbc8cbc931dfe013dde59c111f9332837592a47f257105cb7f24df82
deleted: sha256:1ed2f6427ff9596dd94065719b5d9bc7c5be85c703b4ef0c41ecc30c2fd6c1d6
Deleted build cache objects:
oiyrsimyuz9rmajllipr1zj67
q1bxe5s514pegv9ov8pbeqnre
bphbx4hwwq14hn9t9yrycgfsx
bfrvsoow2dkmjyycjn0tzd2nj
q498s9fcpgc39hv73b12i6k6i
iw0tuvylaw2s24i4qkhzn4kvg
a2lkp9h02y6v87jg4zb81p6ky
efe0f1jjqdkynhbqvjuo1jddy
hyyh8nmvbey95klz2sm9he3nh
rdoyvdti8xd855tb9p0udas7g
qrg97d6dyu3uhg2kqjwmo6fw5
hcuu0x8vy2cc5ebegozgc3z39
Total reclaimed space: 721B
+ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
registry 2 d1fe2eaf6101 2 weeks ago 24.1MB
+ docker build -t localhost:5000/cache-experiments:3 --cache-from localhost:5000/cache-experiments:2 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 51.0s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 229B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye 43.8s
=> importing cache manifest from localhost:5000/cache-experiments:2 0.0s
=> [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 3.6s
=> => resolve docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 0.0s
=> => sha256:bdf0079de4094afdb26b94d9f89b716499436282c972461d945a87899c015c23 1.37kB / 1.37kB 0.0s
=> => sha256:5b7d04484c737c4a2af7f33a85470dab369e922a570411f40ebd72d35167bc49 7.49kB / 7.49kB 0.0s
=> => sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB 1.5s
=> => sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 1.08MB / 1.08MB 0.4s
=> => sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 11.77MB / 11.77MB 1.1s
=> => sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 1.86kB / 1.86kB 0.0s
=> => sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 233B / 233B 0.6s
=> => sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 3.18MB / 3.18MB 1.1s
=> => extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 1.2s
=> => extracting sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 0.1s
=> => extracting sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 0.4s
=> => extracting sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 0.0s
=> => extracting sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 0.2s
=> [internal] load build context 0.0s
=> => transferring context: 368B 0.0s
=> [2/5] WORKDIR /app 0.2s
=> [3/5] COPY requirements.txt ./ 0.0s
=> [4/5] RUN pip install -r requirements.txt 3.0s
=> [5/5] COPY src ./ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7749ebdbcd4d4b5f515a41be6aebd3ceca86d316d952216e6bc350dd0b7ef956 0.0s
=> => naming to localhost:5000/cache-experiments:3 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ rm -rf /mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry
+ docker rm -vf registry
registry
Is it a known issue or something new?
I also noticed that config files which are referred by the manifest.json from the Docker image tarballs are different for the second and the third images. The third one doesn't contain some layers.
I have exactly the same issue, I posted it in another one: https://github.com/moby/buildkit/issues/3009#issuecomment-1260694663
