Cache of COPY --from layers is not working as expected.

[massimeddu-sj]

Actual behavior I'm building a multi-target Docker image like this one:

FROM xxx as builder

COPY ./src. /src

RUN /src/build.sh

FROM yyy as deploy

COPY --from=builder /src/artifact /usr/local/bin/artifact

With the following build options --compressed-caching=true --cache-copy-layers=true --target deploy --reproducible --cache=true --skip-unused-stages=true --cache-run-layers=true --compression=zstd --compression-level=20 --cleanup --snapshot-mode=redo --use-new-run --log-timestamp=true

Now, if the content of the ./src directory is changed the build is re-run in the builder target, but Kaniko is using an (old) cached version of the COPY --from layer in the deploy target, so the new build is not reflecting the changes in the ./src directory.

Expected behavior I expect that the kaniko do not use the cached layer but execute the COPY --from command.

To Reproduce Steps to reproduce the behavior:

1. Create a multi-target Dockerfile, with the first target has a COPY command and the second target has a COPY --from command
2. Run kaniko build with --cache-copy-layers=true option
3. Change the content of the files copied in the first target
4. Re-run kaniko and see kaniko using the wrong cached layer.

N.B: Removing the --cache-copy-layers=true option fixed the issue.

Additional Information Kaniko version 1.12.1

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[aaron-prindle]

Possibly related: https://github.com/GoogleContainerTools/kaniko/pull/2559

[aaron-prindle]

@massimeddu-sonicjobs can you confirm that this is a regression from a previous version of kaniko (issue wasn't in version vX.Y.Z) and if so add the latest version of Kaniko this worked with or is this a bug that has always been the case? Thanks

[massimeddu-sj]

Hi @aaron-prindle , thanks for having a look at this issue. I dug a bit into the codebase, and apparently the root cause of the issue is at: https://github.com/GoogleContainerTools/kaniko/blob/63be4990ca5a60bdf06ddc4d10aa4eca0c0bc714/pkg/executor/composite_cache.go#L79

Basically it looks like that when kaniko calculates the cache key, it excludes the files that are in the .dockerignore file. Now, this is the correct behavior for normal COPY command, but is should not be the case for COPY --from commands.

Actually, removing /src/artifact references from my .dockerfile solved the issue and now the cache key is calculated correctly.

I'm not familiar with go and kaniko codebase, but I'll try to create a small PR to solve the issue in the next few days.

Thank you,

[kt315]

when filename in the .dockerignore docker buildx dont copy to context this file and cant execute COPY .

~/docker_test$ sudo docker buildx build -t blabla .
[+] Building 0.1s (6/7)
 => [internal] load .dockerignore                                         0.0s
 => => transferring context: 49B                                          0.0s
 => [internal] load build definition from Dockerfile                      0.0s
 => => transferring dockerfile: 133B                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04           0.0s
 => [internal] load build context                                         0.0s
 => => transferring context: 2B                                           0.0s
 => [stage1 1/2] FROM docker.io/library/ubuntu:20.04                      0.0s
 => => resolve docker.io/library/ubuntu:20.04                             0.0s
 => ERROR [stage1 2/2] COPY test.txt /                                    0.0s
------
 > [stage1 2/2] COPY test.txt /:
------
error: failed to solve: failed to solve with frontend dockerfile.v0: 
failed to build LLB: failed to compute cache key: "/test.txt" not found: not found




~/docker_test$ tree -a .
.
├── Dockerfile
├── .dockerignore
└── test.txt

0 directories, 3 files



~/docker_test$ cat Dockerfile
FROM ubuntu:20.04 as stage1

COPY test.txt /

FROM ubuntu:20.04

COPY --from=stage1 /test.txt /




~/docker_test$ cat .dockerignore
test.txt

[kt315]

@massimeddu-sonicjobs can you create repo with example project?

[massimeddu-sj]

Hi @kt-315 , sure. Please have a look at this example repo: https://github.com/massimeddu-sonicjobs/kaniko-copy-issue-test

[kt315]

I checked it but dont see mistake. 🤔
When I change test.txt kaniko create new layer in final stage, what am I doing wrong? log here https://pastebin.com/9MHkGT7C

[rgschmitz1]

I'm seeing similar issues of older cache being used even when files have been updated, using Kaniko 1.14.0.

The options I provide are --cache=true --cache-ttl=24h --cache-copy-layers=true --target release --skip-unused-stages=true. I can confirm that disabling --cache-copy-layers does partially resolve the issue but I'd expect layers following a COPY (e.g. RUN commands) to be invalidated (cache miss), but this doesn't appear to be happening.

[kt315]

Hi @rgschmitz1! Can you make example Dockerfile please?

[rgschmitz1]

I can't share my exact source code but I'll try to work out an example, you're going to need a .dockerignore file also.