pack Improve the error message when trying to save a builder in the docker daemon but the architecture doesn't match

Summary

My pack create builder usually runs on GHA (linux/amd64) and I only build for linux/amd64 currently. Over the weekend, I had a weird error:

ERROR: failed to write image to the following tags: [index.docker.io/library/builder:latest: loading image "index.docker.io/library/builder:latest". first error: embedded daemon response: duplicates of file paths not supported]

(This is probably another bug report?)

I tried to run my pack builder create locally, but it fails with:

pack -v builder create builder-unflattened \
		--target linux/amd64 \
		--config builder.toml
ERROR: could not find a target that matches daemon os=linux and architecture=arm64

I've also tried to set DOCKER_DEFAULT_PLATFORM to force it to use linux/amd64 to no avail.

Reproduction

Steps

Create a builder.toml based on the paketo full builder
Run pack

Current behavior

Unable to build on arm64 for amd64.

Expected behavior

I can run pack and create a linux/amd64 builder.

Environment

pack info

Pack:
  Version:  0.38.2+git-f1c347c.build-6533
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.20.11

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13

Config:
  default-builder-image = "[REDACTED]"
  
  [[trusted-builders]]
    name = "[REDACTED]"
  
  [[trusted-builders]]
    name = "[REDACTED]"
  
  [[trusted-builders]]
    name = "[REDACTED]"

docker info--

Client:
 Version:    28.3.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.9.11
    Path:     /Users/till/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.26.1-desktop.1
    Path:     /Users/till/.docker/cli-plugins/docker-buildx
  cloud: Docker Cloud (Docker Inc.)
    Version:  v0.4.18
    Path:     /Users/till/.docker/cli-plugins/docker-cloud
  compose: Docker Compose (Docker Inc.)
    Version:  v2.39.1-desktop.1
    Path:     /Users/till/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.42
    Path:     /Users/till/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.2.0
    Path:     /Users/till/.docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.29
    Path:     /Users/till/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/till/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.13.0
    Path:     /Users/till/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (EXPERIMENTAL) (Docker Inc.)
    Version:  v0.1.36
    Path:     /Users/till/.docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.4.18
    Path:     /Users/till/.docker/cli-plugins/docker-offload
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/till/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.2
    Path:     /Users/till/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/till/.docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /Users/till/.docker/cli-plugins/docker-dev: no such file or directory

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 248
 Server Version: 28.3.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: active
  NodeID: yv6rttgdcqhkkapdzw5wphykr
  Is Manager: true
  ClusterID: an56y9opwacu3u0msggiyy6m7
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.3
  Manager Addresses:
   192.168.65.3:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: 69a280cb-b79b-449a-9492-602276ac80d1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/till/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set

Aug 18 '25 12:08 till

Hi @till

Trying to think about this, I believe the problem is the following: You are trying to build a linux/amd64 builder in a darwin/arm64 machine and save it in your daemon.

Checking your environment information, notice the following:

Pack:
  Version:  0.38.2+git-f1c347c.build-6533
  OS/Arch:  darwin/arm64

The error came from here when pack tries to match the target flag you provided against the storage, in this case, the docker daemon. In your particular case, if you want to build a linux/amd64 in an ARM machine, use --publish flag. You can't save the image to the daemon. Could you try doing that?

pack is not ignoring target flag, but it can't proceed with the request; maybe we could improve the error message

Aug 19 '25 12:08 jjbustamante

So for local builds to test/debug my setup I just can't without publish going through a registry?

Aug 19 '25 12:08 till

So for local builds to test/debug my setup I just can't without publish going through a registry?

I am afraid it is complicated, you are building an linux/amd64 builder based on paketo jammy full builder (which doesn't have arm support yet), and you want to build it and save it locally.

My suggestion will be:

Run a local registry
Use the --publish flag and point to the local registry in your image, for example: http://localhost:5000/my-builder

Aug 19 '25 12:08 jjbustamante

The other way will be, if the builder you are wrapping has arm support, and you can build an arm builder and save it in your daemon, but because you are using paketo, I think that is not an option for you

Aug 19 '25 12:08 jjbustamante

Just for reference, you mentioned

Over the weekend, I had a weird error:

I am not 100% sure, but maybe this error was a side effect of issue #2384 included in pack 0.38

Aug 19 '25 12:08 jjbustamante

Just for reference, you mentioned

Over the weekend, I had a weird error:

I am not 100% sure, but maybe this error was a side effect of issue #2384 included in pack 0.38

I made another issue #2492 for that one. But I am building in CI (GHA) with Linux (amd64).

Debugging that issue let me to this ticket.

I am still confused because I can run and build amd64 locally because Docker supports that. Why can't I build with pack then?

Aug 19 '25 12:08 till

I am afraid it is complicated, you are building an linux/amd64 builder based on paketo jammy full builder (which doesn't have arm support yet), and you want to build it and save it locally.

@jjbustamante - Can you expand on what pack does or cares about to determine if the target is compatible? I was trying to understand this when @till and I were talking yesterday about it and it wasn't clear in the docs. Wondering what specifically is needed here for support?

In a nutshell, a builder is build image + run image + buildpacks (not necessarily images).

There are amd64 and arm64 images for the Paketo full build + run (stack) see.

The reason we don't publish a arm64 builder yet is because some of the buildpacks that support are not published as arm64 yet. Does pack look at/require all of the buildpacks in a builder to have support for the target, i.e. arm64? If so, would filtering the list of buildpacks included in the builder that @till is trying to create to only those with arm64 support be of help?

Aug 19 '25 13:08 dmikusa

For the record: I currently don't need an arm64 build for production.

I wanted to run what we run in CI (because the feedback loop is cumbersome) on my m3 mac (which is arm64).

Aug 19 '25 15:08 till

@dmikusa , @till pack uses imgutil to build the image artifacts (builders, buildpacks or app image), in order to do that we have two main approaches:

When we run pack against the Docker daemon (by default without --publish flag), we use Docker libraries to interact with the daemon; the image saved in the daemon is not the same as the image in a registry. When we build a linux/amd64 image and try to save it in the daemon, we get an error, similar to when you try to do docker pull for something that is not from your architecture. Maybe we could improve the way pack talks with the daemon, but that's the issue in Till's use case, trying to save a linux/amd64 image locally. Probably docker build is doing some other things under the hood to allow that feature, but that's something pack is not doing. Honestly, we don't want to invest a lot of effort in keeping things compatible with the Docker daemon because it is really painful.
When pack runs against a registr, we use https://github.com/google/go-containerregistry, and we can create a standard OCI artifact with its manifest and push it without problems. Because creating a builder and buildpack doesn't require anything special, just layers and a manifest, we build it locally in the file system and then push it.

Aug 21 '25 17:08 jjbustamante

This is really enlightening! Thank you, @jjbustamante

I am guessing buildx would be the way to go, but also what you're saying is that I could use --publish and build for amd64 locally.

A small local registry would suffice then?

I feel the pain with including/maintaining integrations with docker libraries. I'll have a think too if there's a workaround.

Aug 21 '25 17:08 till

A small local registry would suffice then?

Yes! As a workaround, if you just run a local registry, use --publish and push it there, pack should work without any problem

Aug 21 '25 17:08 jjbustamante

Thanks @jjbustamante. I always knew there were differences when using --publish and not using it. This definitely helps to understand that better.

Do you think we could do something more general, like could pack look at the metadata and the current os/arch and maybe output some sort of message if it's being asked to talk to Docker daemon with a questionable target?

Even if it's just like "hey this might work, proceed at your own risk".

The big problem when this happens is that there's no real clues why. If we could have some sort of message that may not always be accurate, but would at least point the user to something about why this might be a problem. That could deflect some of the issues that we're seeing.

Aug 21 '25 18:08 dmikusa