`docker compose build` does not completely reuse cached layers on immediate rebuild, but `docker build` does

[OlaoluwaM]

Description

When attempting to immediately rebuild an arm image for one of our applications, via docker compose build <service> with the following build configuration for the service:

image: ...
build:
      context: ./backend
      dockerfile: multi.dockerfile
      args:
        - GITHUB_TOKEN=${GITHUB_TOKEN}

only the first few layers reuse the cache, the other layers do not for some reason. Here is the Dockerfile:

FROM haskell:9.2.8-buster AS deps
LABEL maintainer="..."
ENV DEBIAN_FRONTEND=noninteractive LANG=C.UTF-8 LC_ALL=C.UTF-8

RUN mkdir -p /opt/build
WORKDIR /opt/build

RUN \
  apt-get update -qq && \
  apt-get install --assume-yes --no-install-recommends \
  libpcre++-dev \
  libpq-dev \
  openssh-client && \
  rm -rf /var/lib/apt/lists/*

# `docker compose build` will make use of cached layers until it reaches this layer
RUN stack update # cache cabal index update in a layer

# ...Rest of dockerfile

However, when we attempt this with docker build instead, caching working as expected, that is, during an immediate rebuild, all layers are reused from the cache.

It's only when we specify the cache_from and cache_to options in the build configuration of the service that caching works as expected with docker compose build:

image: ...
build:
      context: ./backend
      dockerfile: multi.dockerfile
      args:
        - GITHUB_TOKEN=${GITHUB_TOKEN}
      cache_from:
        - type=local,src=/path/to/cache/dir
      cache_to:
        - type=local,dest=/path/to/cache/dir

Perhaps there might be something going on with the default cache for docker compose?

Reproduce

1. Run docker compose build <service>
2. Wait for image to finish building
3. Run docker compose build <same-service> again
4. Notice that only the first few layers are reused from the cache

Expected behavior

On an immediate rebuild with docker compose build all layers should be reused from the cache without having to specify cache_from and cache_to in the service build configuration

docker version

Client:
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:26:02 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.31.0 (153195)
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:12 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.33
  GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.1.4
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1-desktop.1
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1-desktop.1
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.24
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.9.3
    Path:     /Users/ola.mustapha/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 26.1.4
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.31-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 11.67GiB
 Name: docker-desktop
 ID: 8cebdd5b-de86-4dcd-9a36-32395c41f802
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/ola.mustapha/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

E6294795-5A03-4FF6-AC19-C23E7DD47C65/20240628103937

Additional Info

This is on Macos Sonoma (v14.5) with an M2 Pro chip