for-mac
for-mac copied to clipboard
docker
`com.apple.provenance` extended file attribute busts remote layer cache from `cache-from`
Description
When building an image with buildx build and --cache-from (to utilize remote caching), when it gets to a COPY instruction it is unable to utilize cached layers, even though the files being copied have the same file content as what exists in the image. In the real world, we are pushing this cache image on Github runners running Ubuntu, but I have shared a reproducer script that you can run on your laptop.
I believe this happens because files on my machine have a com.apple.provenance extended file attribute on them. In my research, there are numerous other com.apple.* file attributes that can also easily bust remote layer caching, as the values are all semi-unique. This is especially problematic in orgs that utilize remote layer caching as technique to improve local development experience. Because of this, there are many more Docker builds than what should be required.
Reproduce
# clear all cache
docker system prune -a -f && docker buildx prune -f
cat << EOF > Dockerfile
FROM alpine:latest
WORKDIR /app
COPY package.json ./
# simulate a long build
RUN sleep 10
RUN date > /app/date.txt
EOF
cat << EOF > package.json
{
"name": "test",
"version": "1.0.0",
"main": "index.js"
}
EOF
# take note of the xattr values
xattr -lxv package.json
# something like:
# package.json: com.apple.provenance:
# 00000000 01 02 00 85 20 64 69 12 1C 7A 7D |.... di..z}|
# 0000000B
# build and push cache image
docker buildx build --platform linux/arm64 --push \
--cache-to type=registry,ref=acme/some-image:alpine-cache,mode=max,image-manifest=true,oci-mediatypes=true \
--cache-from acme/some-image:alpine-cache \
--output type=registry,name=acme/some-image:alpine .
# clear all cache
docker system prune -a -f && docker buildx prune -f
# build again to test that the cache is used. Note, this does not push the image.
docker buildx build --platform linux/arm64 \
--cache-from acme/some-image:alpine-cache \
--output acme/some-image:alpine-latest .
# this should be cached like this:
# [+] Building 1.5s (12/12) FINISHED docker:desktop-linux
# => [internal] load build definition from Dockerfile 0.0s
# => => transferring dockerfile: 193B 0.0s
# => [internal] load metadata for docker.io/library/alpine:latest 0.5s
# => [internal] load .dockerignore 0.0s
# => => transferring context: 2B 0.0s
# => importing cache manifest from acme/some-image:alpine-cache 0.6s
# => => inferred cache manifest type: application/vnd.oci.image.manifest.v1+json 0.0s
# => [1/5] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s
# => => resolve docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s
# => [internal] load build context 0.0s
# => => transferring context: 141B 0.0s
# => [auth] sharing credentials for ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com 0.0s
# => CACHED [2/5] WORKDIR /app 0.0s
# => CACHED [3/5] COPY package.json ./ 0.0s
# => CACHED [4/5] RUN sleep 10 0.0s
# => [5/5] RUN date > /app/date.txt 0.3s
# => => sha256:6e710e64083f773817526a54ad4a59597d358d8cd21037b496b7e37f5f17e38e 159B / 159B 0.1s
# => => sha256:6a4c6a5a989aa50b45761f398921de747dc641f8f947c3a5207f663cf410e2b6 190B / 190B 0.2s
# => => sha256:c5699d9eddca2e1c0e976683c0a6d07f1077eee5ef404d207c7a0bd4d277629a 93B / 93B 0.2s
# => => extracting sha256:c5699d9eddca2e1c0e976683c0a6d07f1077eee5ef404d207c7a0bd4d277629a 0.0s
# => => extracting sha256:6a4c6a5a989aa50b45761f398921de747dc641f8f947c3a5207f663cf410e2b6 0.0s
# => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s
# => => extracting sha256:6e710e64083f773817526a54ad4a59597d358d8cd21037b496b7e37f5f17e38e 0.0s
# => exporting to client directory 0.3s
# => => copying files 8.20MB 0.1s
# clear all cache
docker system prune -a -f && docker buildx prune -f
# open the 'Terminal.app' (Terminal.app is the only terminal capable of deleting xattr values) and run:
xattr -c package.json
# this should remove the xattr values
xattr -lxv package.json
# this should now be empty
# (back in your original terminal) build again to test that the cache is used
docker buildx build --platform linux/arm64 \
--cache-from acme/some-image:alpine-cache \
--output acme/some-image:alpine-latest .
# This is not cached!!
# [+] Building 12.0s (13/13) FINISHED docker:desktop-linux
# => [internal] load build definition from Dockerfile 0.0s
# => => transferring dockerfile: 193B 0.0s
# => [internal] load metadata for docker.io/library/alpine:latest 0.7s
# => [auth] library/alpine:pull token for registry-1.docker.io 0.0s
# => [internal] load .dockerignore 0.0s
# => => transferring context: 2B 0.0s
# => importing cache manifest from acme/some-image:alpine-cache 0.7s
# => => inferred cache manifest type: application/vnd.oci.image.manifest.v1+json 0.0s
# => [1/5] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s
# => => resolve docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s
# => [internal] load build context 0.0s
# => => transferring context: 104B 0.0s
# => [auth] sharing credentials for ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com 0.0s
# => [2/5] WORKDIR /app 0.1s
# => => sha256:c5699d9eddca2e1c0e976683c0a6d07f1077eee5ef404d207c7a0bd4d277629a 93B / 93B 0.1s
# => => extracting sha256:c5699d9eddca2e1c0e976683c0a6d07f1077eee5ef404d207c7a0bd4d277629a 0.0s
# => [3/5] COPY package.json ./ 0.0s
# => [4/5] RUN sleep 10 10.2s
# => [5/5] RUN date > /app/date.txt 0.2s
# => exporting to client directory 0.1s
# => => copying files 8.20MB 0.1s
Expected behavior
The remote image layer cache should be used because only this one file attribute has changed, but the file content is still the same.
IMO, Docker should ignore these com.apple.* file attributes because they are very likely to invalidate any performance gains from remote layer caching.
docker version
Client:
Version: 28.0.1
API version: 1.48
Go version: go1.23.6
Git commit: 068a01e
Built: Wed Feb 26 10:38:16 2025
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.39.0 (184744)
Engine:
Version: 28.0.1
API version: 1.48 (minimum version 1.24)
Go version: go1.23.6
Git commit: bbd0a17
Built: Wed Feb 26 10:40:57 2025
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 28.0.1
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Docker AI Agent - Ask Gordon (Docker Inc.)
Version: v0.9.4
Path: /Users/user.name/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.1-desktop.2
Path: /Users/user.name/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.1-desktop.1
Path: /Users/user.name/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.38
Path: /Users/user.name/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Beta) (Docker Inc.)
Version: v0.1.5
Path: /Users/user.name/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/user.name/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: /Users/user.name/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/user.name/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/user.name/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/user.name/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.16.3
Path: /Users/user.name/.docker/cli-plugins/docker-scout
Server:
Containers: 15
Running: 14
Paused: 0
Stopped: 1
Images: 11
Server Version: 28.0.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 14
Total Memory: 23.43GiB
Name: docker-desktop
ID: d02e4181-4b3f-4b8d-b176-4df5521cfad4
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 187
Goroutines: 201
System Time: 2025-03-25T19:50:55.112029673Z
EventsListeners: 13
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/user.name/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
::1/128
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
A4724C0C-1F40-4DEA-9DD7-1CAABA3DB291/20250325195138
Additional Info
While typing up this issue, I came across a few pages that discuss this com.apple.provenance file attribute.
- https://apple.stackexchange.com/questions/478422/why-do-iterm2-vscode-on-macos-sequoia-keep-adding-com-apple-provenance-to-my/478430#478430
- https://eclecticlight.co/2023/03/13/ventura-has-changed-app-quarantine-with-a-new-xattr/
This attribute is only added to files when you create/modify a file using a non-builtin MacOS app. I am able to confirm that when I do this in 'Terminal.app':
- clear the attribute :
xattr -c package.json - check that the attribute is gone:
xattr -lxv package.json - touch 'package.json':
touch package.json - check the file again to see if there are any attributes again (there aren't):
xattr -lxv package.json
However, when I do this again in 'ITerm2' (also using 'Terminal.app' to remove the attribute) the com.apple.provenance attribute is added once again.
- clear the attribute in 'Terminal.app':
xattr -c package.json - check that the attribute is gone:
xattr -lxv package.json - (in ITerm2) touch 'package.json':
touch package.json - check the file again to see that the attribute is added back:
xattr -lxv package.json
My current laptop is an M4 Macbook, which is new to me. My old M1 Macbook does not have this attribute on every file like my new laptop, even though I use ITerm2 on both computers. Both have SIP enabled (csrutil status), and both are running the same version of Docker/Docker Desktop.
I've found a workaround that fixes this caching issue for me. After performing these steps, cache images built in Github runners work correctly locally.
- Add your terminal, and editors to the 'Developer Tools' list under 'Privacy & Security > Developer Tools'. I've also added Docker for good measure.
-
Restart your computer
-
Run
xattr -d com.apple.provenance -r * && xattr -d com.apple.provenance -r .* && xattr -d com.apple.provenance -r .to remove this file attribute in the directory where your build context lives. I did this in the root of my repo locally. -
Run
xattr -lxvr * && xattr -lxvr .*to check that the attribute is gone
Note, there might be other attributes that exist on your files that effect caching, so you will need to review and remove them.
I'm on an M3 mac, but can confirm this workaround allows me to utilize the cache built on ubuntu github runners. However, in my case, I had to disable system integrity protection in order to fully remove the attributes.
@tonistiigi I may be looking in the wrong place, but as far as I can tell, the only place where xattrs are filtered out when calculating layer checksums is here
if k == "security.capability" || !strings.HasPrefix(k, "security.") && !strings.HasPrefix(k, "system.") {
xAttrKeys = append(xAttrKeys, k)
}
Would it make sense to also exclude com.apple.provenance? Or potentially com.apple.*?
Would it make sense to also exclude com.apple.provenance? Or potentially com.apple.*?
More likely these xattrs should never be copied to a Linux environment at all.
Does COPY --chown x:x --chmod x work around this issue?
edit: Doesn't seem like it :(
edit 2: Allowing my applications to use developer tools didn't work around the issue either. I will try again complete with system reboot shortly
edit 3: I misread the workaround as "restart application" rather than "restart computer". After restart and removing attributes, the workaround worked as expected.
Would it make sense to also exclude com.apple.provenance? Or potentially com.apple.*?
More likely these xattrs should never be copied to a Linux environment at all.
@tonistiigi Do you know which component is responsible for that? Is it the Docker for Mac Desktop client or something else?
@tonistiigi I'm not sure if this is the correct place for a fix, but I opened up a PR in buildkit to try and address this.