docker build --no-cache uses cache anyway :'(

[gummyWalrus]

Description

I'm trying to build an image referencing a custom image that references maven. I'm using docker cli on linux Fedora 36

Here is the dockerfile

FROM lanico/whanos-java:latest
WORKDIR /app
COPY . .
WORKDIR /app/app
RUN mvn package
RUN ls -R /app
COPY /app/app/target/app.jar .
CMD ["java" , "-jar", "app.jar"]

the image referenced by this Dockerfile is built using this one.

FROM maven:3.8.5-openjdk-17

I successfully built and push the referenced one by using

docker build --no-cache -t lanico/whanos-java:latest
docker push lanico/whanos-java:latest

But then when I try to build the first one "the referencer" It uses cache anyway

docker build --no-cahce --pull -t lanico/test-java:latest

Output :

$ docker build --no-cache --pull -t lanico/test-java:latest .

[+] Building 1.6s (12/12) FINISHED                                                                                                                                                                    
 => [internal] load build definition from Dockerfile                                                                                                                                             0.0s
 => => transferring dockerfile: 203B                                                                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                                                                0.1s
 => => transferring context: 2B                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/lanico/whanos-java:latest                                                                                                                             1.4s
 => [auth] lanico/whanos-java:pull token for registry-1.docker.io                                                                                                                                0.0s
 => [internal] load build context                                                                                                                                                                0.1s
 => => transferring context: 3.81kB                                                                                                                                                              0.0s
 => [1/7] FROM docker.io/lanico/whanos-java:latest@sha256:2ed711842f59534bac58dc1834c678068011a611ac53aefcded028d533c05e4b                                                                       0.1s
 => => resolve docker.io/lanico/whanos-java:latest@sha256:2ed711842f59534bac58dc1834c678068011a611ac53aefcded028d533c05e4b                                                                       0.1s
 => CACHED [2/7] WORKDIR /app                                                                                                                                                                    0.0s
 => CACHED [3/7] COPY . .                                                                                                                                                                        0.0s
 => CACHED [4/7] WORKDIR /app/app                                                                                                                                                                0.0s
 => CACHED [5/7] RUN mvn package                                                                                                                                                                 0.0s
 => CACHED [6/7] RUN ls -R /app                                                                                                                                                                  0.0s
 => ERROR [7/7] COPY /app/app/target/app.jar .                                                                                                                                                   0.0s
------
 > [7/7] COPY /app/app/target/app.jar .:
------
Dockerfile:7
--------------------
   5 |     RUN mvn package
   6 |     RUN ls -R /app
   7 | >>> COPY /app/app/target/app.jar .
   8 |     CMD ["java" , "-jar", "app.jar"]
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref moby::zceqi6bcku5ggj2hte8gg96i0: "/app/app/target/app.jar": not found

Docker has cached the failing mvn package from a previous build and now, it is not building the target/app.jar :'(

It's not even running my ls command.

I've tried to

docker system prune
docker builder prune --all

still NOT working :(

Reproduce

1. docker build --no-cache -t
2. See that it uses cache :'(

Expected behavior

docker build --no-cache should not show me CACHED in the output. And should not use CACHE (how does it uses cache when I'm removing it prior to build !)

docker version

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:50:04 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:47:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 75
 Server Version: 23.0.1
 Storage Driver: btrfs
  Btrfs: 
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.11-100.fc36.x86_64
 Operating System: Fedora Linux 36 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.643GiB
 Name: fedora
 ID: QFM4:EZPH:JWCQ:ZUQ5:AMV4:4FOL:OSD5:V35N:EHZM:FBLX:QP5M:WBEI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: lanico
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Linux kernel version : Linux fedora 6.1.11-100.fc36.x86_64