Docker CLI Hangs for all commands

[dvndrsn]

trafficstars

Description

Docker CLI hangs when running any command. Fresh MacOS and Docker install on M1 Mac.

MacOS: 13.5 (22G74) Chip: Apple M1 Pro Docker Desktop: 4.22.0 (117440)

Tried re-installing / factory resetting docker and clearing docker files as suggested in this issue: https://github.com/docker/for-mac/issues/6527

Self diagnosis shows that VM cannot be started. Any tips for next steps?

[email protected]@Dave's-Mac ~ % /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[FAIL] DD0017: can a VM be started? vm has not started: failed to open kmsg.log: open log/vm/kmsg.log: no such file or directory
[FAIL] DD0016: is the LinuxKit VM running? prereq failed: can a VM be started?
[FAIL] DD0011: are the LinuxKit services running? prereq failed: is the LinuxKit VM running?
[FAIL] DD0004: is the Docker engine running? prereq failed: are the LinuxKit services running?
[PASS] DD0015: are the binary symlinks installed?
[FAIL] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[PASS] DD0013: is the $PATH ok?
[FAIL] DD0003: is the Docker CLI working? prereq failed: is the Docker engine running?
[FAIL] DD0038: is the connection to Docker working? prereq failed: is the Docker engine running?
[FAIL] DD0014: are the backend processes running? prereq failed: is the LinuxKit VM running?
[FAIL] DD0007: is the backend responding? prereq failed: are the backend processes running?
[PASS] DD0008: is the native API responding?
[FAIL] DD0009: is the vpnkit API responding? prereq failed: are the backend processes running?
[FAIL] DD0010: is the Docker API proxy responding? prereq failed: are the backend processes running?
[FAIL] DD0030: is the image access management authorized? prereq failed: is the Docker engine running?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[WARN] DD0017: can a VM be started? vm has not started: failed to open kmsg.log: open log/vm/kmsg.log: no such file or directory
[WARN] DD0016: is the LinuxKit VM running? prereq failed: can a VM be started?
[WARN] DD0011: are the LinuxKit services running? prereq failed: is the LinuxKit VM running?
[WARN] DD0004: is the Docker engine running? prereq failed: are the LinuxKit services running?
[PASS] DD0015: are the binary symlinks installed?
[WARN] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[WARN] DD0032: do Docker networks overlap with host IPs? prereq failed: does the Docker API work?

Please note the following 6 warnings:

1 : The check: can a VM be started?
    Produced the following warning: vm has not started: failed to open kmsg.log: open log/vm/kmsg.log: no such file or directory

The Docker engine runs inside a Linux VM. Therefore we must be able to start Virtual Machines.

2 : The check: is the LinuxKit VM running?
    Produced the following warning: prereq failed: can a VM be started?

The Docker engine runs inside a Linux VM. Therefore the VM must be running.

3 : The check: are the LinuxKit services running?
    Produced the following warning: prereq failed: is the LinuxKit VM running?

The Docker engine runs inside a Linux VM as a service. Therefore the services must have started.

4 : The check: is the Docker engine running?
    Produced the following warning: prereq failed: are the LinuxKit services running?

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.

5 : The check: does the Docker API work?
    Produced the following warning: prereq failed: is the Docker engine running?

If the Docker API is not available from the host then Docker Desktop will not work correctly.

6 : The check: do Docker networks overlap with host IPs?
    Produced the following warning: prereq failed: does the Docker API work?

If the subnet used by a Docker network overlaps with an IP used by the host, then containers
won't be able to contact the overlapping IP addresses.

Try configuring the IP address range used by networks: in your docker-compose.yml.
See https://docs.docker.com/compose/compose-file/compose-file-v2/#ipv4_address-ipv6_address


Please investigate the following 1 issue:

1 : The test: can a VM be started?
    Failed with: vm has not started: failed to open kmsg.log: open log/vm/kmsg.log: no such file or directory

The Docker engine runs inside a Linux VM. Therefore we must be able to start Virtual Machines.

Reproduce

1. Run any command - docker ps, docker info, etc.

Expected behavior

Command should output as expected (ex. list of processes or info)

docker version

Running `docker version` hangs on CLI.

% docker --version
Docker version 24.0.5, build ced0996

docker info

Hangs, no output.

Diagnostics ID

CDA29737-2B38-4A57-9384-068DB58FFD94/20230809183121

Additional Info

First diagnostic from docker desktop: CDA29737-2B38-4A57-9384-068DB58FFD94/20230809183121 Ran another diagnostic from the command line: CDA29737-2B38-4A57-9384-068DB58FFD94/20230809183808

[ctalledo]

Hi @dvndrsn, sorry to hear you hit a problem but thanks for uploading the diagnostics bundle, that really helps us.

I looked at the logs in the diagnostics bundle, and I can see that the Docker Desktop VM fails to come up, though it's not exactly clear why (we have some hints in there though).

Question: was this a one-time failure, or does it reoccur frequently with Docker Desktop v4.22?

Thanks and apologies again for the issue.

[MuzzamielBarrows]

Having the same issue on mac m1

[ctalledo]

Having the same issue on mac m1

Hi @MuzzamielBarrows: if possible, could you upload a diagnostics bundle please?

Thanks!

[dvndrsn]

Question: was this a one-time failure, or does it reoccur frequently with Docker Desktop v4.22?

I've not yet been able to run docker successfully on this machine.

I can see that the Docker Desktop VM fails to come up, though it's not exactly clear why (we have some hints in there though)

Hints? Please share! I can see from the self-diangostic (great tool btw) that the Linux VM isn't coming up, but I'm not clear what intervention I can do on my end to try to restart that VM and help it along..

[kvasylchenko]

I have exactly the same on my Mac with M2 chip.

MacOS: 13.5 Chip: Apple M2 Pro Docker Desktop: 4.22.0 (117440)

[VerstraeteBert]

Same issue here on M1, MacOs 13.5, Docker Desktop 4.22. Identical diagnosis report.

[ctalledo]

Thanks all for filing the reports.

We have a new (not yet released) build with a potential fix.

If you could give it a try and let us know if it fixes the problem (or not), we would much appreciate it.

Thanks!

[dvndrsn]

Update: I installed Docker Desktop 4.21.1 from the release notes page and simple commands like docker info and docker version no longer hang.

There must be a problem specifically with 4.22.0 and Apple Silicon (at least for fresh installs?).

[ctalledo]

Thanks @dvndrsn.

To provide more context, in DD v4.22 we have a new resource saver feature that turns off the Docker Desktop Linux VM whenever Docker goes unused for a period of time (e.g., 30 secs or more). The goal is to reduce host memory utilization (by 2GB+), whenever you are not using Docker.

This feature works well for most users, but a few have reported issues with Docker Desktop startup as well as exit from resource saver mode.

We have an unreleased build with a potential fix, if you could try it and let us know if it works, we would much appreciate it.

Thanks again!

[dvndrsn]

Thanks @ctalledo! I will try the unreleased build once I've gotten my environment up and running with the prior release of docker for mac.

[guillaumebl]

Hello, I've got the same issue and I tried upgrading to v4.23. Sadly it did not solve the issue, the only way to fix it has been to downgrade to 4.21

[ctalledo]

Hello, I've got the same issue and I tried upgrading to v4.23. Sadly it did not solve the issue, the only way to fix it has been to downgrade to 4.21

Thanks @guillaumebl for letting us know; would you have a diagnostics bundle that we can look at (ideally with v4.23)?

[adrianwix]

I have the same issue even on 4.21.1 (114176) sometimes docker just stops working

Apple M2 macOS 13.5

Then I have restart the computer, delete all my imagines and volumes and pray the gods it works. You can imagine the drop in productivity

I can't even update it

[guillaumebl]

@ctalledo Here's the Diagnostics ID : BA6F13F0-E794-489B-80A5-26CF0E0A7E9C/20230817155545 Thanks

[ickeundso]

I'm not sure this my problem is the same but seems to be in that direction. I can run DD 4.22 in Mac M1 / MacOS 13.5 . Everthing works fine. But after a wile , several minutes / sometime hours . All containers are hanging (except my nginx container is delivering a http 502) errors) and all docker commands like docker ps hanging too. I can open the DD UI and can see 400% for one CPU but ( normally if it works I have configured 6CPU's and is show as such after restart) But no running container is shown. I attach the /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check docker-stuck-completely.log A restart from the DD-UI brings the containers and the docker commands to work again. So I'm going to disable the [resource saver feature] , I will see then.

[ickeundso]

Ok, also when the Experimental feature for resource saving is deselected in the DD-UI , the engine will stop after a while. But on the other hand on my MacBook Pro with M1 , with no running container . Docker is running and also a e.g docker ps will respond with the right values.

[ickeundso]

It seems as if version 4.22 doesn't cause issues on every M1 device. On my MacBook: M1 Max 64GB, Docker runs without problems, both with "Resource Saver" enabled and with or without running containers. For example, "docker ps" can still be executed even after hours. However, on my Mac mini M1 16GB, it's different. Whether "Resource Saver" is enabled or disabled, Docker stops working shortly after some time. Unfortunately, I can't pinpoint exactly when this happens, but it's not too long. My Mac isn't connected to a monitor; I only access it via SSH or, if necessary, through Remote Desktop.So, after downgrading to version 4.21.1 on my Mac mini and keeping the running containers unchanged, same behaviour as with 4.22 .

[ickeundso]

Happens also with 4.20.0 on my mac min, after around 1h: Support Id: B0B019C8-0DE1-452F-AC20-31E518C7CDA6/20230820160627

Not only the "docker SOMETHING" does not work also a "Quit Docker Desktop" doesn't. But the "Restart" on Docker Desktop was working on 4.22. And after that I can also use "Quit Docker Desktop" or everything else. But with 4.20 I have to restart the machine or to kill this docker ' docker serve --address unix:///Users/maex/.docker/run/docker-cli-api.sock' from the shell.

[ickeundso]

one more difference between my MacBook M1 and the Mac mini. The diagnostic check on the MacBook has no issues but on then Mac mini I get every time: `[FAIL] DD0014: are the backend processes running? 1 error occurred: * com.docker.vpnkit is not running

[PASS] DD0007: is the backend responding? [PASS] DD0008: is the native API responding? [FAIL] DD0009: is the vpnkit API responding? dial unix vpnkit.diag.sock: connect: connection refused`

can this be related to this issue?

[ickeundso]

So after downgrading to 4.18 and after disable Use Virtualization framework and using gRPC for file sharing. Docker works for 2 days without any problem. So then I updated (per DD-UI) to 4.22 again an checked that Use Virtualization framework is still disabled. The problem occurs again after a few hours. B0B019C8-0DE1-452F-AC20-31E518C7CDA6/20230823053837

[ickeundso]

So now I tried to go forward to find out what version breaks the docker stability. But I need no much time. 4.19 worked perfect but with 4.20 the docker hangs again after a short time <2h . Here the diagnose Id for 4.20 : B0B019C8-0DE1-452F-AC20-31E518C7CDA6/20230825065317

after the fresh installation on my mac mini: ` /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check [2023-08-25T07:35:40.903328000Z][com.docker.diagnose][I] set path configuration to OnHost Starting diagnostics

[PASS] DD0027: is there available disk space on the host? [PASS] DD0028: is there available VM disk space? [PASS] DD0018: does the host support virtualization? [PASS] DD0001: is the application running? [PASS] DD0017: can a VM be started? [PASS] DD0016: is the LinuxKit VM running? [PASS] DD0011: are the LinuxKit services running? [PASS] DD0004: is the Docker engine running? [PASS] DD0015: are the binary symlinks installed? [PASS] DD0031: does the Docker API work? [PASS] DD0013: is the $PATH ok? [PASS] DD0003: is the Docker CLI working? [PASS] DD0038: is the connection to Docker working? [PASS] DD0014: are the backend processes running? [PASS] DD0007: is the backend responding? [PASS] DD0008: is the native API responding? [SKIP] DD0009: is the vpnkit API responding? [PASS] DD0010: is the Docker API proxy responding? [SKIP] DD0030: is the image access management authorized? [PASS] DD0033: does the host have Internet access? [PASS] DD0018: does the host support virtualization? [PASS] DD0001: is the application running? [PASS] DD0017: can a VM be started? [PASS] DD0016: is the LinuxKit VM running? [PASS] DD0011: are the LinuxKit services running? [PASS] DD0004: is the Docker engine running? [PASS] DD0015: are the binary symlinks installed? [PASS] DD0031: does the Docker API work? [PASS] DD0032: do Docker networks overlap with host IPs? No fatal errors detected.`

and after <2 h running until I figure out that the containers not longer reachable and the command a hanging again: produces then :

`[PASS] DD0027: is there available disk space on the host? [PASS] DD0028: is there available VM disk space? [PASS] DD0018: does the host support virtualization? [PASS] DD0001: is the application running? [PASS] DD0017: can a VM be started? [PASS] DD0016: is the LinuxKit VM running? [FAIL] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:12:11.085836000Z][com.docker.diagnose][I] ipc.NewClient: 9359e6b3-diagnose -> diagnosticd.sock diagnosticsd [2023-08-25T07:12:11.085981000Z][com.docker.diagnose][I] (9d3f2e99) 9359e6b3-diagnose C->S diagnosticsd GET /ping [2023-08-25T07:12:12.088495000Z][com.docker.diagnose][W] (9d3f2e99) 9359e6b3-diagnose C<-S NoResponse GET /ping (1.002422792s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[FAIL] DD0004: is the Docker engine running? Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:12:12.089228000Z][com.docker.diagnose][I] ipc.NewClient: ad7c7947-com.docker.diagnose -> lifecycle-server.sock VMDockerdAPI [2023-08-25T07:12:12.089315000Z][com.docker.diagnose][I] (7054c471) ad7c7947-com.docker.diagnose C->S VMDockerdAPI GET /docker [2023-08-25T07:17:12.098781000Z][com.docker.diagnose][W] (7054c471) ad7c7947-com.docker.diagnose C<-S NoResponse GET /docker (5m0.002550833s): Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:17:12.100408000Z][com.docker.diagnose][I] (7054c471-1) ad7c7947-com.docker.diagnose C->S VMDockerdAPI GET /ping [2023-08-25T07:17:13.102354000Z][com.docker.diagnose][W] (7054c471-1) ad7c7947-com.docker.diagnose C<-S NoResponse GET /ping (1.001904958s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:17:14.103689000Z][com.docker.diagnose][I] (7054c471-2) ad7c7947-com.docker.diagnose C->S VMDockerdAPI GET /ping [2023-08-25T07:17:15.105425000Z][com.docker.diagnose][W] (7054c471-2) ad7c7947-com.docker.diagnose C<-S NoResponse GET /ping (1.001625416s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:17:16.106867000Z][com.docker.diagnose][I] (7054c471-3) ad7c7947-com.docker.diagnose C->S VMDockerdAPI GET /ping [2023-08-25T07:17:17.108166000Z][com.docker.diagnose][W] (7054c471-3) ad7c7947-com.docker.diagnose C<-S NoResponse GET /ping (1.001270208s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [2023-08-25T07:17:18.109422000Z][com.docker.diagnose][I] (7054c471-4) ad7c7947-com.docker.diagnose C->S VMDockerdAPI GET /ping [2023-08-25T07:17:19.111342000Z][com.docker.diagnose][W] (7054c471-4) ad7c7947-com.docker.diagnose C<-S NoResponse GET /ping (1.001871417s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[PASS] DD0015: are the binary symlinks installed? [FAIL] DD0031: does the Docker API work? Get "http://docker.raw.sock/v1.24/containers/json": context deadline exceeded [PASS] DD0013: is the $PATH ok? [FAIL] DD0003: is the Docker CLI working? signal: killed [PASS] DD0038: is the connection to Docker working? [PASS] DD0014: are the backend processes running? [PASS] DD0007: is the backend responding? [PASS] DD0008: is the native API responding? [SKIP] DD0009: is the vpnkit API responding? [PASS] DD0010: is the Docker API proxy responding? [SKIP] DD0030: is the image access management authorized? [PASS] DD0033: does the host have Internet access? [PASS] DD0018: does the host support virtualization? [PASS] DD0001: is the application running? [PASS] DD0017: can a VM be started? [PASS] DD0016: is the LinuxKit VM running? [WARN] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [WARN] DD0004: is the Docker engine running? Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [PASS] DD0015: are the binary symlinks installed? [WARN] DD0031: does the Docker API work? Get "http://docker.raw.sock/v1.24/containers/json": context deadline exceeded [WARN] DD0032: do Docker networks overlap with host IPs? Get "http://docker.raw.sock/v1.24/networks": context deadline exceeded

Please note the following 4 warnings:

1 : The check: are the LinuxKit services running? Produced the following warning: failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The Docker engine runs inside a Linux VM as a service. Therefore the services must have started.

2 : The check: is the Docker engine running? Produced the following warning: Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.

3 : The check: does the Docker API work? Produced the following warning: Get "http://docker.raw.sock/v1.24/containers/json": context deadline exceeded

If the Docker API is not available from the host then Docker Desktop will not work correctly.

4 : The check: do Docker networks overlap with host IPs? Produced the following warning: Get "http://docker.raw.sock/v1.24/networks": context deadline exceeded

If the subnet used by a Docker network overlaps with an IP used by the host, then containers won't be able to contact the overlapping IP addresses.

Try configuring the IP address range used by networks: in your docker-compose.yml. See https://docs.docker.com/compose/compose-file/compose-file-v2/#ipv4_address-ipv6_address

Please investigate the following 1 issue:

1 : The test: are the LinuxKit services running? Failed with: failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The Docker engine runs inside a Linux VM as a service. Therefore the services must have started.`

[ctalledo]

Hi @ickeundso, could you please try with Docker Desktop v4.22.1? If you hit the issue again, please upload the diagnostics bundle for this version. Thanks!

[ickeundso]

Hi @ctalledo, this first try didn't work. I did an update in DD from 4.18 -> 4.22.1 (unfortunately I didn't get a diagnostic). So now I'm trying another approche. I went back to 4.18 , run this version without issue for 24h. So now stopped the DD . Installed the new version 4.22.1 from a download file and restarted my host machine. For now, it works for at least 12h. This is longer than in most cases described above. So I will wait for another couple of hours. Hopefully it will work again in the same reliable way as I know docker for years.

[ickeundso]

Hi, @ctalledo , so it seems to work for me. For 2 day docker 4.22.1 run without an problem. All my containers and the docker commands are functional and working as expected. I also enabled "Use virtialisation Framwork" and VirtioFs. I do not konw if someone need this, but also for this successfully 2 days run I uploaded the diagnostics. B0B019C8-0DE1-452F-AC20-31E518C7CDA6/20230903071459. Thank you for investigating and fixing the Issue!

[0x2d3c]

v4.22.1 desktop have the same question, in cmd docker ps -a with nothing out put

[serdar]

I have the same issue as well on an M2 MacBook Pro running 4.22.1. (Mac OS version 13.4.1 (c) )

Diagnostics ID: FCA5E48B-4E08-4800-B63F-71AC0585560A/20230907060156

[ctalledo]

Hi @serdar,

I have the same issue as well on an M2 MacBook Pro running 4.22.1. (Mac OS version 13.4.1 (c) ) Diagnostics attached.

Thanks for reporting the issue. When you run the diagnostics, Docker Desktop should have uploaded them and you should have received a diagnostics ID; could you share that ID number so I can pull it up? The serdar-docker.log file you attached is not sufficient for me to debug.

Thanks!

[adamczykpiotr]

Same thing for Docker (24.0.2-rd, build e63f5fa) + Rancher desktop on M2 Pro. For me it hangs only for specific operations - i.e. with composer and then running docker compose run --rm composer install. After this docker becomes unoperable, docker ps doesn't respond.

[0x2d3c]

v4.22.1 desktop have the same question, in cmd docker ps -a with nothing out put

i fix this question by downgrade to v4.20, mac mini m1

[0x2d3c]

I have the same issue as well on an M2 MacBook Pro running 4.22.1. (Mac OS version 13.4.1 (c) )

Diagnostics ID: FCA5E48B-4E08-4800-B63F-71AC0585560A/20230907060156

maybe downgrade to 4.20 can fix your question