Packaging a multiarch Docker image fails with `ERROR: docker exporter does not currently support exporting manifest lists`

[d-m]

Describe the bug Trying to package a multiarch Docker image fails with:

ERROR: docker exporter does not currently support exporting manifest lists

Pants version 2.21.0

OS MacOS

Docker version Using Rancher Desktop 1.15.1

Client:
 Version:           27.1.1-rd
 API version:       1.45 (downgraded from 1.46)
 Go version:        go1.21.12
 Git commit:        cdc3063
 Built:             Wed Jul 24 17:06:24 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server:
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.5
  Git commit:       8e96db1c328d0467b015768e42a62c0f834970bb
  Built:            Sun Jul  7 17:34:20 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.7.17
  GitCommit:        3a4de459a68952ffb703bbe7f2290861a75b6b67
 runc:
  Version:          1.1.12
  GitCommit:        51d5e94601ceffbbd85688df1c928ecccbfa4685
 docker-init:
  Version:          0.19.0
  GitCommit:  

Additional info pants.toml

[docker]
use_buildx = true
tools = ["docker-credential-osxkeychain"]

BUILD

docker_image(
    name="myimage",
    build_platform=["linux/amd64", "linux/arm64"],
)

Error in logs:

10:01:24.03 [DEBUG] Completed: setup_sandbox
10:01:24.04 [DEBUG] spawned local process as Some(90884) for Process { argv: ["/Users/me/.rd/bin/docker", "buildx", "build", "--platform=linux/amd64,linux/arm64", "--output=type=docker", "--pull=False", "--tag", "myimage:latest", "--file", "src/myimage/docker/Dockerfile", "."], env: {"PATH": "/private/var/folders/rl/pb4_f0gx7jd2rv72hsntsqcr0000gn/T/pants-sandbox-VmZMy1/_binary_shims_2fde2be74164a49d18304f1390149d066360229ea09f63c11ca1cf5a509fe222", "__UPSTREAM_IMAGE_IDS": ""}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<10a68159516c914fa1e41d951d6dc9385d7ac42b576de3d60f6a95d61e912d6a>, size_bytes: 323 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<13d6d975ac8d9000e912d8749a7d355aaf5d15c63f4cf8d0cefa469879959435>, size_bytes: 170 }, tree: "Some(..)" }, immutable_inputs: {RelativePath("_binary_shims_2fde2be74164a49d18304f1390149d066360229ea09f63c11ca1cf5a509fe222"): DirectoryDigest { digest: Digest { hash: Fingerprint<2fde2be74164a49d18304f1390149d066360229ea09f63c11ca1cf5a509fe222>, size_bytes: 105 }, tree: "Some(..)" }}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Building docker image myimage:latest", level: Info, append_only_caches: {}, jdk_home: None, cache_scope: PerSession, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
10:01:24.31 [INFO] Completed: Building docker image myimage:latest
10:01:24.31 [DEBUG] Completed: Scheduling: Building docker image myimage:latest
10:01:24.31 [DEBUG] Completed: `package` goal
10:01:24.31 [DEBUG] computed 1 nodes in 2.673538 seconds. there are 988 total nodes.
10:01:24.31 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building docker image myimage:latest' failed with exit code 1.
stdout:

stderr:
ERROR: docker exporter does not currently support exporting manifest lists



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

This appears to be related to https://github.com/docker/buildx/issues/59.

[huonw]

Sorry for the trouble. To make easier to debug, is it possible to create a fully-standalone reproducer? That is, share the full pants.toml you're using?

[d-m]

Sure, here it is:

[GLOBAL]
pants_version = "2.21.0"

backend_packages = [
    "pants.backend.codegen.protobuf.lint.buf",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.docker",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.experimental.docker.podman",
    "pants.backend.experimental.python.lint.ruff.check",
    "pants.backend.experimental.python.lint.ruff.format",
    "pants.backend.python",
    "pants.backend.python.typecheck.mypy",
]

ca_certs_path = "./cert.pem"

[test]
use_coverage = true
report = true

[coverage-py]
report = ["console", "xml"]
global_report = true

[docker]
env_vars = ["DOCKER_CERT_PATH=./cert.pem"]
use_buildx = true
tools = ["docker-credential-osxkeychain"]

[python]
interpreter_constraints = ["CPython==3.12.5"]
enable_resolves = true
default_resolve = "python-default"

[python.resolves]
python-default = "3rdparty/python/default.lock"
mypy = "3rdparty/python/mypy.lock"

[mypy]
install_from_resolve = "mypy"

[source]
root_patterns = ["docker", "protobuf", "python"]

[export]
py_editable_in_resolve = ["python-default"]

[python-bootstrap]
search_path = ["<PYENV_LOCAL>", "<PYENV>", "<ASDF>", "<PATH>"]

[d-m]

I was able to get the build to work by connecting to the Rancher Desktop VM with rdctl shell, changing the daemon.json to use the containerd-snapshotter:

cat <<EOF >/etc/docker/daemon.json
{
  "features": {
    "containerd-snapshotter": true
  }
}
EOF

and then restarting dockerd.

I think the issue has to do with the flag "--output=type=docker" that Pants is using, which corresponds to --load in https://github.com/docker/buildx/issues/59.

In order for the build to work with the default snapshotter it looks like it might have to be broken up into a multistep build and then load as shown at https://github.com/docker/buildx/issues/59#issuecomment-616050491.

[jyang-gentex]

Solution: https://github.com/docker/buildx/issues/59#issuecomment-2612971318