`docker images -a` default sort (by {{.CreatedAt}}) is unstable for images created at the same second

[ryan-williams]

Description docker images -a prints images in reverse-chronological order based on their creation time.

For images whose creation timestamp falls in the same wall-clock second, the order that images are printed is non-deterministic / unstable.

Steps to reproduce the issue:

1. Make this Dockerfile:

FROM alpine:3.12.0
WORKDIR /bin
WORKDIR /
WORKDIR /usr
WORKDIR /usr/bin
WORKDIR /var

2. Build it:

docker build .

3. Repeatedly ask docker images for the most recent few images, observe different orders for images with the same (second-granularity) timestamp, e.g.:

docker images -a '--format={{.ID}} {{.CreatedAt}}' | head -n 5

2097ad743e52 2020-07-18 20:02:54 +0000 UTC
4f5bf8cfa447 2020-07-18 20:02:53 +0000 UTC
aa3a1f10f414 2020-07-18 20:02:53 +0000 UTC
028722166df4 2020-07-18 20:02:53 +0000 UTC
13ac996787b0 2020-07-18 20:02:53 +0000 UTC

docker images -a '--format={{.ID}} {{.CreatedAt}}' | head -n 5

2097ad743e52 2020-07-18 20:02:54 +0000 UTC
028722166df4 2020-07-18 20:02:53 +0000 UTC
aa3a1f10f414 2020-07-18 20:02:53 +0000 UTC
4f5bf8cfa447 2020-07-18 20:02:53 +0000 UTC
13ac996787b0 2020-07-18 20:02:53 +0000 UTC

Alternatively, run successive invocations of docker images -a through diff:

images() {
  docker images -a '--format={{.ID}} {{.CreatedAt}}' | head -n 5
}
diff <(images) <(images)

Describe the results you received: Order of returned images is nondeterministic.

Describe the results you expected: Order of returned images to be deterministic.

Additional information you deem important (e.g. issue happens only occasionally): I've only reproduced it with the -a flag, which includes untagged / intermediate images.

I didn't try to create tagged images at the same second, to see if the issue is present with docker images as well, though I suspect it is.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 302
  Running: 1
  Paused: 0
  Stopped: 301
 Images: 992
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 11.71GiB
 Name: docker-desktop
 ID: 6YW6:GPHF:GA5P:DBFL:SBLS:YKKQ:SC2J:5RBY:SD53:BXHU:Z2YH:PQD7
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 46
  Goroutines: 56
  System Time: 2020-07-18T20:11:43.7306303Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.): macOS Catalina 10.15.5

[eiffel-fl]

Hi.

I can reproduce it. But I think the problem does not come from docker-cli but from below. Indeed, with:

images() {
  curl --unix-socket /var/run/docker.sock 'http://localhost/images/json?all=true' | jq '.'
}
diff <(images) <(images)

There is the same problem.

I took a quick look at containerd code and the images list seems to be created in ListImages. With a deeper look seems, I think the list creation is related to protobuf with List. So maybe the best place to do a workaround is in ListImages

Best regards.