compose
compose copied to clipboard
docker
[BUG] platform flag in compose.yaml file STILL ignored for M3 mac
Description
This error was supposed to have been fixed from reading other tickets... but its still not working for me. I've tried several versions of Docker Desktop, including the latest (4.27.2). This is an image that starts up fine with docker run --platform=linux/amd64. However a docker compose .yaml file (that worked on non-M3 mac) with
services:
- platform: linux/amd64
- ......
does not work. I have built the image with docker build --platform=linux/amd64 .... and this hasnt made a difference. This is a major issue for our applications that need to run/manage local images through Docker Desktop for M3 macs. I've also tried specifying DOCKER_DEFAULT_PLATFORM=linux/amd64 before the docker-compose command and no luck. Is this bug planning to be fixed... or is there a workaround for this that I havent found through naive google search?
Steps To Reproduce
- mac M3 machine
- docker desktop 4.27.2
- docker-compose -f ~/chemistream/lib/python3.10/site-packages/makalii-1.0.0-py3.10.egg/makalii/setup_files/docker-compose-slurmMaster.yaml up -d (the yaml file is any file to start a docker image)
- docker image specified in *.yaml is built with --platform=/linux/amd64
Output says image has started... but docker ps --all shows the container as having exiting
Compose Version
phoenix2:chemistream-hpc-images swsides$ docker-compose version
Docker Compose version v2.24.5-desktop.1
phoenix2:chemistream-hpc-images swsides$ docker compose version
Docker Compose version v2.24.5-desktop.1
Docker Environment
phoenix2:chemistream-hpc-images swsides$ docker info
Client:
Version: 25.0.3
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1-desktop.4
Path: /Users/swsides/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.5-desktop.1
Path: /Users/swsides/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container. (Docker Inc.)
Version: 0.0.24
Path: /Users/swsides/.docker/cli-plugins/docker-debug
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/swsides/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: /Users/swsides/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: /Users/swsides/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.0.0
Path: /Users/swsides/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/swsides/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.4.1
Path: /Users/swsides/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/swsides/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/swsides/.docker/cli-plugins/docker-scan: no such file or directory
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.6.12-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 7.657GiB
Name: docker-desktop
ID: 5fc6af4e-114c-479e-914b-04ebd6e03956
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
Anything else?
No response
I tried to reproduce (on a Mac M1) but don't get any issue here is my reproduction scenario:
$ cat compose.yaml
services:
test:
image: toto
platform: linux/amd64
$ cat Dockerfile
FROM alpine
CMD ping localhost
$ docker build --platform linux/amd64 -t toto .
[+] Building 0.6s (5/5) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 68B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.6s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/1] FROM docker.io/library/alpine:latest@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1b10476d2267bb79a5b4f924c163a6126d2691d2a1e844dd10a2995b6272e6c5 0.0s
=> => naming to docker.io/library/toto 0.0s
$ docker inspect toto -f '{{.Architecture}}'
amd64
$ docker compose run test
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.126 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.606 ms
I hardly understand how M3 would have any impact as I'm already not running with the host' plaform. Can you please confirm you can reproduce this issue with my basic example ?
Yes... I dont understand how M3 could affect it either ;-) I'll give your test a try... thanks
I cant really share the docker image build files. They are proprietary and complicated (multi-stage etc). I can tell you they are built off of a centos7 image.
This is the docker compose file I'm using
services: spec: shm_size: '5gb' privileged: true tty: true ports: - '8901:9000' - '6006:6006' - '6007:6007' - '6008:6008' - '6009:6009' image: registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images/csimg-nfp_qpredict:latest labels: - "com.txcorp.hpccloud.SessionTitle=tech-x-swsides_QPredict" - "com.txcorp.hpccloud.SessionImage=QPredict" - "com.txcorp.hpccloud.SessionOwner=tech-x-swsides" volumes: - /Users/swsides/chemistream_shared_dir:/home/hpcuser/cluster_shared_storage environment: - ENV_SSHD_PORT=1022 - ENV_USE_INTERFACE=en0 - ENV_SCHEDULER=slurm - ENV_SLURM_CONF_PATH=/etc/slurm/slurm.conf - ENV_MUNGE_KEY_PATH=/etc/munge/munge.key
Your simple test does work on my M3. But as you can see from my compose yaml file above... I set tty true and some others that are different from your compose.yaml file (not sure if these are the issues). Also... here's the startup command
docker compose -f docker-compose-slurmMaster.yaml up -d
so i'm detaching from the container as well.
Can you please run docker inspect on container created by compose, which has stopped ?
Please confirm platform is well set, and capture exit code / state
please note this was already the case with v1 version of the parser, we just restored this restriction
Not sure what you mean by 'capture exit code / state'. But here's output from docker-compose up startup
phoenix2:~ swsides$ docker-compose -f docker-compose-slurmMaster.yaml up -d
[+] Running 1/1
✔ Container swsides-spec-1 Started 0.1s
phoenix2:~ swsides$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8b46ea59b79 phoenix/rlmolecule "/usr/sbin/initd" 7 seconds ago Exited (1) 6 seconds ago swsides-spec-1
Here's the output of inspect.
phoenix2:~ swsides$ docker inspect d8b46ea59b79
[
{
"Id": "d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779",
"Created": "2024-02-21T18:44:38.253634833Z",
"Path": "/usr/sbin/initd",
"Args": [],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2024-02-21T18:44:38.383659792Z",
"FinishedAt": "2024-02-21T18:44:38.419134875Z"
},
"Image": "sha256:00bdc42eb3f4face6ba6d01a757ce3ceb019042f94788e4cc5031c2f49a31b6f",
"ResolvConfPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/hostname",
"HostsPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/hosts",
"LogPath": "/var/lib/docker/containers/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779/d8b46ea59b79a7aad54b53b47435306a8ac81b162e0b7aeb08db8059f758e779-json.log",
"Name": "/swsides-spec-1",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/Users/swsides/chemistream_shared_dir:/home/hpcuser/cluster_shared_storage:rw"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "swsides_default",
"PortBindings": {
"6006/tcp": [
{
"HostIp": "",
"HostPort": "6006"
}
],
"6007/tcp": [
{
"HostIp": "",
"HostPort": "6007"
}
],
"6008/tcp": [
{
"HostIp": "",
"HostPort": "6008"
}
],
"6009/tcp": [
{
"HostIp": "",
"HostPort": "6009"
}
],
"9000/tcp": [
{
"HostIp": "",
"HostPort": "8901"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 5368709120,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc-init/diff:/var/lib/docker/overlay2/939577f790b65522dabb4f5e9ab746a2f4d916ed42f49bbb99678ee7c1e1acb1/diff",
"MergedDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/merged",
"UpperDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/diff",
"WorkDir": "/var/lib/docker/overlay2/a7b29e5a8552410f6daa19c64e9cef63971948fb7657adaf703866c82d5a3bcc/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/Users/swsides/chemistream_shared_dir",
"Destination": "/home/hpcuser/cluster_shared_storage",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "d8b46ea59b79",
"Domainname": "",
"User": "root",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"6006/tcp": {},
"6007/tcp": {},
"6008/tcp": {},
"6009/tcp": {},
"9000/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"ENV_MUNGE_KEY_PATH=/etc/munge/munge.key",
"ENV_SSHD_PORT=1022",
"ENV_USE_INTERFACE=en0",
"ENV_SCHEDULER=slurm",
"ENV_SLURM_CONF_PATH=/etc/slurm/slurm.conf",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"ENV_PRODCUT=default",
"ENV_BATCH_SCRIPT_ENTRYPOINT=/usr/local/bin/user-batch.sh",
"CONDA_BIN=/home/hpcuser/cs-env/bin",
"PYTHONDONTWRITEBYTECODE=true",
"LD_LIBRARY_PATH=:/home/hpcuser/cs-env/lib"
],
"Cmd": [
"/usr/sbin/initd"
],
"Image": "phoenix/rlmolecule",
"Volumes": null,
"WorkingDir": "/home/hpcuser",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"build-date": "20170911",
"build.by": "ci",
"build.commit": "77275f1f076fefd0f4de4d3ff08a703b1270a600",
"build.number": "2069",
"build.target.environment": "staging",
"build.timestamp": "2020-02-19T14:02:47",
"com.docker.compose.config-hash": "30bc297d9e844d1c8c8e3fbaa08d1850cef5dbb077fdabca522bad5e647fbbe3",
"com.docker.compose.container-number": "1",
"com.docker.compose.depends_on": "",
"com.docker.compose.image": "sha256:00bdc42eb3f4face6ba6d01a757ce3ceb019042f94788e4cc5031c2f49a31b6f",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "swsides",
"com.docker.compose.project.config_files": "/Users/swsides/docker-compose-slurmMaster.yaml",
"com.docker.compose.project.working_dir": "/Users/swsides",
"com.docker.compose.service": "spec",
"com.docker.compose.version": "2.24.5",
"com.theubercloud.compose": "IyDCqSAyMDEzIC0gMjAxOSwgVGhlVWJlckNsb3VkLCBJbmMuIEFsbCByaWdodHMgcmVzZXJ2ZWQuCgpuYW1lOiBjZW50b3NfNy4xCmluZm86CiAgdGl0bGU6IENlbnRPUwogIHZlcnNpb246IDcuMQogIGRlc2NyaXB0aW9uOiBVYmVyQ2xvdWQgYmFzZSBzeXN0ZW0gb24gQ2VudE9TIDcuMQogIHByb3ZpZGVyOiBVYmVyQ2xvdWQKc3BlYzoKICBwcml2aWxlZ2VkOiB0cnVlCiAgcG9ydHM6CiAgICAgICAgICAtICIxMDIyOiR7RU5WX1NTSERfUE9SVH0iCiAgI2RuczogW10KICBlbnZ2YXJzOgogICAgLSBFTlZfU1NIRF9QT1JUPTIyCiAgICAgICMtIEVOVl9PUkRFUl9OVU1CRVI9CiAgICAgICMtIEVOVl9DTFVTVEVSX0RJU0NPVkVSWT1jb25zdWw6Ly88U0VSVkVSX0lQPjo4NTAwCiAgICAgICMtIEVOVl9ST0xFPXNsYXZlCiAgICAgICMtIEVOVl9VU0VfSU5URVJGQUNFPWV0aDAKICAgICAgIy0gRU5WX0lORkxVWERCX1VSTD1odHRwOi8vMTcyLjE3LjAuMTo4MDg2Cg==",
"com.txcorp.hpccloud.SessionImage": "QPredict",
"com.txcorp.hpccloud.SessionOwner": "tech-x-swsides",
"com.txcorp.hpccloud.SessionTitle": "tech-x-swsides_QPredict",
"license": "GPLv2",
"name": "CentOS Base Image",
"vendor": "CentOS"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "3aa7e96dc6956d8483c1408b9291981b8997b53d0604f6c308edc47b9a36b9f7",
"SandboxKey": "/var/run/docker/netns/3aa7e96dc695",
"Ports": {},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"swsides_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"swsides-spec-1",
"spec",
"d8b46ea59b79"
],
"MacAddress": "",
"NetworkID": "4a72725704acb01d4f0bfbe4501ba3f2e2d30e65b1d51ce6d1b22629d384129a",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"swsides-spec-1",
"spec",
"d8b46ea59b79"
]
}
}
}
}
]
I just realized M3 is arm64... not amd64. However I'm still seeing the container exit immediately. So just to make sure
- I'm running on mac M3 (arm64)
- My image is built on linux centos7
- I'm building the image with docker build --platform=linux/arm64 .....
- I'm running with docker run --platform=linux/arm64 ....
Generally this should be working right?
Also... I gave up on docker-compose and tried simply starting with docker run and specifying --tty and --interactive and that is exiting immediately as well. So its maybe not a docker compose issue. Do I need to simply give up on M3 and docker?
Yes this should work, if requested platform is not supported compose up should not succeed to create container.
Can you please confirm you successfully can run this same image using a plain docker run command ? inspect output suggest container was well created, but exited with code 1 (did you checked docker logs xxx in case there's something reported there?)
The log only has the following info:
phoenix2:chemistream-hpc-images swsides$ docker logs e6b45848f5e9 2024/02/22 16:51:10 Cannot get container ID
As I mentioned, the docker run command is not working either. The command is
docker run
--privileged
--detach
--platform=linux/arm64/v8
phoenix/rlmolecule
I believe linux/arm64/v8 is the right architecture... but I could be wrong. Querying the platform with python gives
import platform platform.uname() uname_result(system='Darwin', node='phoenix2.local', release='23.2.0', version='Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:51 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6030', machine='x86_64')
I thought linux/arm64/v8 was the correct architecture because some error messages were saying that was what docker was autodetecting, but I could be wrong.
Also, I can start an interactive session in a container. Its only when I want to detach from the container that it exits immediately. Very confusing... I've never seen this behavior before.
the docker run command is not working either.
ok so this isn't a docker compose issue ! You should report on github.com/docker/cli
Also, I can start an interactive session in a container. Its only when I want to detach from the container that it exits immediately. Very confusing... I've never seen this behavior before.
Which command do you use to run an interactive session?
Yes... I believe this is a docker thing. Now if I add "-i" to the command... then I can detach and leave the container running. This has never been the case in the past.
I'm still testing but what I suspect is this: only the later versions of Docker will run on M3 (i think >4.24.0). So on M3 I need new version of docker.... and this version has subtle differences in behavior with certain options.
Ok... after more testing I think these are the essential steps. The image MUST be built with --platform=linux/arm64/v8. Then all is needed is to add entrypoint: /bin/bash to the compose.yaml file. Then with simple docker-compose -f compose.yaml up -d the same behavior is recovered. Again, I suspect that some kind of default entrypoint behavior in older versions of docker was changed in more recent versions.
can you reproduce running docker run -it --entrypoint /bin/bash xxx?
could you also please try to use a minimalist Dockefile (just FROM alpine for example) to confirm this is a runtime and not image design issue?
Please report to github.com/moby/moby with a reproduction example
Sorry, by my statement "... same behavior is recovered... " I mean the desired behavior of the image staying up. So... I believe I've fixed the problem. I dont know WHY the changes I describe above are needed... my only guess is that the behavior of certain options in docker changes subtly over time and this latest version of Docker that can install on M3 simply has sligthly different behavior.