buildx panic in buildx imagetools create

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description

Trying to retag a multi arch image with the following approach:

docker buildx imagetools inspect --raw registry1/image:tag \
  | docker buildx imagetools create -f /dev/stdin -t registry2/image:tag

Expected behaviour

A new multi arch manifest should be created that is identical to the source one registry1/image:tag except for its name registry2/image:tag

Actual behaviour

+ docker buildx imagetools inspect --raw registry1/image:tag
+ docker buildx imagetools create -f /dev/stdin -t registry2/image:tag
[+] Building 1.2s (0/1)                                                                                                                 
 => [internal] pushing docker.io/registry2/image:tag                                                              1.2s
 => => # pushing  to docker.io/registry2/image:tag                                                                    
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d815f6]

goroutine 36 [running]:
io.(*PipeWriter).Write(...)
	io/pipe.go:165
github.com/containerd/containerd/remotes/docker.(*pushWriter).Commit(0xc00033cb60, {0x0?, 0x32bad80?}, 0x0, {0x0, 0x0}, {0xc144ad620ea4a840?, 0x12479a348?, 0x4408a80?})
	github.com/containerd/[email protected]/remotes/docker/pusher.go:441 +0x76
github.com/containerd/containerd/content.Copy({0x32de2b8, 0xc000782120}, {0x32e2968, 0xc00033cb60}, {0x32ba940, 0xc000798f60}, 0x0, {0x0, 0x0}, {0x0, ...})
	github.com/containerd/[email protected]/content/helpers.go:186 +0x3df
github.com/docker/buildx/util/imagetools.(*Resolver).Push(0x1052847?, {0x32de210?, 0xc000466320?}, {0x32cc040, 0xc000358f00}, {{0xc000245500, 0x27}, {0x0, 0x0}, 0x0, ...}, ...)
	github.com/docker/buildx/util/imagetools/create.go:175 +0x29c
github.com/docker/buildx/commands/imagetools.runCreate.func2.1({0x32cd8a0?, 0xc0006a4150})
	github.com/docker/buildx/commands/imagetools/create.go:200 +0x4d0
github.com/docker/buildx/util/progress.Wrap({0xc0006ae1c0, 0x36}, 0xc0006a4078, 0xc00071ce90)
	github.com/docker/buildx/util/progress/progress.go:47 +0x2dc
github.com/docker/buildx/commands/imagetools.runCreate.func2()
	github.com/docker/buildx/commands/imagetools/create.go:183 +0x29a
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

Buildx version

github.com/docker/buildx v0.11.2-desktop.5 f20ec1393426619870066baba9618cf999063886

Docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/tsuna/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/tsuna/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/tsuna/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/tsuna/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/tsuna/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/tsuna/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/tsuna/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/tsuna/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 27
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 5.801GiB
 Name: docker-desktop
 ID: af734301-00ac-40de-be69-c52ed71f6d72
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 44
  Goroutines: 68
  System Time: 2023-10-20T08:51:46.222447261Z
  EventsListeners: 11
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false



### Builders list

```text
NAME/NODE       DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default         docker                                       
  default       default         running v0.11.6+616c3f613b54 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux * docker                                       
  desktop-linux desktop-linux   running v0.11.6+616c3f613b54 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6



### Configuration

docker buildx imagetools inspect --raw registry1/image:tag
| docker buildx imagetools create -f /dev/stdin -t registry2/image:tag


### Build logs

_No response_

### Additional info

_No response_

Oct 20 '23 08:10 tsuna

docker buildx imagetools inspect --raw registry1/image:tag \
  | docker buildx imagetools create -f /dev/stdin -t registry2/image:tag

Thanks for your report, this should not panic but we don't currently support reading a source from stdin. You can achieve the same behavior with:

docker buildx imagetools create registry1/image:tag -t registry2/image:tag

Oct 20 '23 09:10 crazy-max

That worked, thanks!

Oct 20 '23 09:10 tsuna

Do you have a version of the output from docker buildx imagetools inspect --raw registry1/image:tag that I could look at? Sanitized input is fine. I just want to know the shape.

My primary focus is just on getting it not to panic.

Oct 20 '23 19:10 jsternberg

@jsternberg I repro with alpine:

$ docker buildx imagetools inspect --raw alpine | docker buildx imagetools create -f /dev/stdin -t crazymax/alpine:test-imagetools-stdin
#1 [internal] pushing docker.io/crazymax/alpine:test-imagetools-stdin
#1 0.000 pushing  to docker.io/crazymax/alpine:test-imagetools-stdin
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11c0825]

goroutine 83 [running]:
io.(*PipeWriter).Write(...)
        io/pipe.go:165
github.com/containerd/containerd/remotes/docker.(*pushWriter).Commit(0xc0006bea10, {0x0?, 0x259a3e0?}, 0x0, {0x0, 0x0}, {0xc144d49a4f0fdda9?, 0x98e55e8d?, 0x36e9360?})
        github.com/containerd/[email protected]/remotes/docker/pusher.go:452 +0x65
github.com/containerd/containerd/content.Copy({0x25bdaa0, 0xc00044dc80}, {0x25c3c08, 0xc0006bea10}, {0x259a740, 0xc000988ae0}, 0x0, {0x0, 0x0}, {0x0, ...})
        github.com/containerd/[email protected]/content/helpers.go:186 +0x3b8
github.com/docker/buildx/util/imagetools.(*Resolver).Push(0xc00044d980, {0x25bdad8?, 0xc00064e000?}, {0x25a6818, 0xc000458f00}, {{0xc000121b40, 0x39}, {0x0, 0x0}, 0x0, ...}, ...)
        github.com/docker/buildx/util/imagetools/create.go:210 +0x409
github.com/docker/buildx/commands/imagetools.runCreate.func2.1({0x25aca60?, 0xc000881338})
        github.com/docker/buildx/commands/imagetools/create.go:202 +0x457
github.com/docker/buildx/util/progress.Wrap({0xc000055500, 0x37}, 0xc0008812f0, 0xc000407e90)
        github.com/docker/buildx/util/progress/progress.go:47 +0x2b9
github.com/docker/buildx/commands/imagetools.runCreate.func2()
        github.com/docker/buildx/commands/imagetools/create.go:185 +0x285
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x96

It seems to also happen on latest containerd 1.7.7 in https://github.com/containerd/containerd/blob/788f7f248a1ea5cbc3ffd5fa1be0fa3511dfe4fe/remotes/docker/pusher.go#L448. Seems like pipe writer is nil.

Oct 20 '23 19:10 crazy-max

Looks like a combo of a couple issues:

It looks like if Write is not called, then when Commit is called, pipe will be nil. Is it possible that no content is actually being written, and it's trying to create an empty file? Other functions on the pushWriter seem to have nil checks for this scenario, but Commit does not.

Note: this looks similar in style/spirit to https://github.com/containerd/containerd/pull/8379/.
The expected argument to buildx imagetools create isn't a manifest/manifest list - it's a descriptor: https://github.com/docker/buildx/blob/7838ade9f369b164a405628a37e8dedbe91bdd10/commands/imagetools/create.go#L240-L267

Maybe we should also support manifest/manifest lists? Or at the very least, validate the output to ensure we get a reasonable descriptor (the error check there only fails on bad JSON syntax, not missing fields).

Oct 23 '23 11:10 jedevc

The second is what I was noticing. There seems to be a section of the code that handles different descriptor media types, but some of them seem to have their content just get discarded by that function. I was going to try to trace it out a bit more today.

At the least, we should identify that the argument is wrong. But we presently just unmarshal the JSON and go c'est la vie to whatever the result is.

Oct 23 '23 15:10 jsternberg