docker-compose down --rmi=local: unrecognized image

[x-yuri]

Description

docker-compose down --rmi=local occasionally fails if an image is shared by several services.

Steps to reproduce the issue:

docker-compose.yml:

services:
  s1:
    build: .
  s2:
    build: .

Dockerfile:

FROM alpine

$ set -x && for i in {1..10}; do docker-compose up >/dev/null 2>&1 && docker-compose down --rmi=local; done
...
+ docker-compose up
+ docker-compose down --rmi=local
[+] Running 4/5
 ⠿ Container tmpq37b5off7h-s1-1   Removed                                                                                 0.0s
 ⠿ Container tmpq37b5off7h-s2-1   Removed                                                                                 0.0s
 ⠙ Image tmpq37b5off7h_s1         Removing                                                                                0.1s
 ⠿ Network tmpq37b5off7h_default  Removed                                                                                 0.1s
 ⠿ Image tmpq37b5off7h_s2         Removed                                                                                 0.0s
Error response from daemon: unrecognized image ID sha256:16a98022286ca55293845f2f506ae822f4588be116c93544f40ef4ce271c696f
...

At least a couple times out of 10 it fails for me.

Describe the results you received: It outputs an error. Although it supposedly succeeded.

Describe the results you expected: It finishes w/o an error.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version 2.6.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., 2.6.0)

Server:
 Containers: 102
  Running: 0
  Paused: 0
  Stopped: 102
 Images: 1145
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.18.5-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.02GiB
 Name: yuri2
 ID: GDZF:WOJX:D4NW:H545:6ACG:PLQT:M3CQ:4QKD:TD2V:T5LW:HSBG:QHBC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

[milas]

The underlying issue here appears to be a potential race in the Docker engine API - I can reproduce this on v2.10.2 consistently within 10 iterations as well.

There were some improvements around image cleanup in #9819 in v2.11.0, which incidentally appear to avoid triggering this, and I can't reproduce it anymore.

@x-yuri Would you be able to test on the latest release (v2.11.0) and let me know if you still see it?

---

If you do, I've got a fix ready to go (posting here for posterity):

func isImageNotFoundErr(err error) bool {
	if err == nil {
		return false
	}

	// see https://github.com/moby/moby/blob/924edb948c2731df3b77697a8fcc85da3f6eef57/image/store.go#L228-L235
	return errdefs.IsNotFound(err) || strings.Contains(err.Error(), "unrecognized image")
}

[milas]

Closing as I believe Compose changes prevent triggering the upstream issue. See moby/moby#44290 for the underlying engine/API bug.

[x-yuri]

It appears I've somehow missed your previous message :( I can't reproduce it either with docker-compose 2.11.x. Not within 100 iterations.