docker
Containerd snapshotter does not save layer chain id info in metadata.db for locally built image for platform other than host platform using docker buildx
Description
We had built an image for platform linux/arm64 using docker buildx on amd64 machine. We were unable to see the layer chainID information for this image in metadata.db from where we could find the linkage to the layer directory number in snapshots folder for that layer. How should we corelate the layer chain ID info with snapshot directory number then?
Also when we do docker rmi -f ImageID , it does not remove the layer contents from content/blobs/sha256 folder also not from overlayfs/snapshots folder.
Reproduce
Build an image using Dockerfile docker buildx build --platform linux/arm64 -t my-image . docker images --tree (fetch the layer from config of the image) open the metadata.db from /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db and search for respective layer chainID
Expected behavior
We were expecting the image layers chainID info in metadata.db before pushing it to registry just same as we get after we pull it from registry if same image is pushed.
docker version
[root@OEL8 ~]# docker version
Client: Docker Engine - Community
Version: 26.1.0
API version: 1.45
Go version: go1.21.9
Git commit: 9714adc
Built: Mon Apr 22 17:07:54 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.0
API version: 1.45 (minimum version 1.24)
Go version: go1.21.9
Git commit: c8af8eb
Built: Mon Apr 22 17:06:47 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
[root@OEL8 ~]#
docker info
[root@OEL8 ~]# docker info
Client: Docker Engine - Community
Version: 26.1.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.26.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 7
Server Version: 26.1.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 5.4.17-2136.330.7.1.el8uek.x86_64
Operating System: Oracle Linux Server 8.9
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.544GiB
Name: OEL8
ID: 1bdb9cd9-009f-4f78-85f8-19324984ea0d
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http://10.113.194.54:3128
HTTPS Proxy: http://10.113.194.54:3128/
No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
Username: fanny5798
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
[root@OEL8 ~]#
Additional Info
we have tried on docker 27 too still the issue exists there.
This is likely better to be opened as a discussion in either https://github.com/moby/buildkit/discussions or https://github.com/moby/moby/discussions, because this repository is for the Docker CLI, which is only a client for the daemon.
That said, /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db is an internal / implementation of containerd, and not intended to be accessed directly. Perhaps if you open a discussion you could provide details about what you're trying to achieve; what do you need the information for?
Thank you for the reply. Also, how about the clean up of the image layers from /var/lib/containerd/ after we do docker -rmi? It does not clean up and its a manual process.