for-mac Docker hangs on Disk Image Location change to empty folder on external volume

[x] I have tried with the latest version of Docker Desktop
[x] I have tried disabling enabled experimental features
[x] I have uploaded Diagnostics
Diagnostics ID: A86CC52E-36D4-4F96-B334-E7F775C2A92D/20230413123832

Expected behavior

After changing Settings -> Resources -> Advanced -> Disk Image location to external volume -> empty folder, Docker should successfully change it to new location and restart.

Actual behavior

After changing Settings -> Resources -> Advanced -> Disk Image location to an empty folder on external volume, Docker hangs forever on "Apply & restart", the circle of progress bar constantly spins. Docker service becomes inactive and hangs in restarting state. Screenshot 2023-04-13 at 16 26 53

If I try to do it for the same partition, where disk image originally was, it works as intended.

Information

I did a clean install of Docker Desktop from the official site. Waited for it to finish changing directory for more than 30 mins. The directory to where I move is clean of any files. A disk space available is more than 100Gb on both drives, where Docker is installed and the directory, where I move. The disk is free of errors and the both volumes for Docker and where I move is internal disk of MacBook, which never failed and allows blazing fast disk access. I also tried:

Restart Docker or Quit and then start over with checking its not running in Activity Monitor
Reboot and apply the setting again many times
Checked the rights on the dir @ external drive, so I able to move files here, it's not write-protected
No antivirus, at Little Snitch firewall Docker has full access to any outbound/inbound connection
Fully reinstall with uninstaller /Applications/Docker.app/Contents/MacOS/uninstall and reboot for sure before installing again
Stop Docker service and then change the dir to another
Reduce disk limit and swap to the minimum values available, 8Gb for disk limit and 0Gb for swap respectively.
- macOS Version: Ventura 13.2.1
- Intel chip or Apple chip: Apple M1
- Docker Desktop Version: 4.18.0

Output of `/Applications/Docker.app/Contents/MacOS/com.docker.diagnose check`

[2023-04-13T12:40:46.973584000Z][com.docker.diagnose][I] set path configuration to OnHost
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[FAIL] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": dial unix diagnosticd.sock: connect: no such file or directory
[2023-04-13T12:40:46.983219000Z][com.docker.diagnose][I] ipc.NewClient: 69cbf4f4-diagnose -> diagnosticd.sock diagnosticsd
[2023-04-13T12:40:46.983437000Z][com.docker.diagnose][I] (f5619988) 69cbf4f4-diagnose C->S diagnosticsd GET /ping
[2023-04-13T12:40:46.983746000Z][com.docker.diagnose][W] (f5619988) 69cbf4f4-diagnose C<-S NoResponse GET /ping (303.834µs): Get "http://ipc/ping": dial unix diagnosticd.sock: connect: no such file or directory

[FAIL] DD0004: is the Docker engine running? Get "http://ipc/docker": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:46.984240000Z][com.docker.diagnose][I] ipc.NewClient: 3ad77a33-com.docker.diagnose -> lifecycle-server.sock VMDockerdAPI
[2023-04-13T12:40:46.984564000Z][com.docker.diagnose][I] (4db55475) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /docker
[2023-04-13T12:40:46.984871000Z][com.docker.diagnose][W] (4db55475) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /docker (304.875µs): Get "http://ipc/docker": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:46.985083000Z][com.docker.diagnose][I] (4db55475-1) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:46.985309000Z][com.docker.diagnose][W] (4db55475-1) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (224.709µs): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:47.985670000Z][com.docker.diagnose][I] (4db55475-2) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:47.988156000Z][com.docker.diagnose][W] (4db55475-2) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.485708ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:48.989218000Z][com.docker.diagnose][I] (4db55475-3) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:48.991239000Z][com.docker.diagnose][W] (4db55475-3) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.127042ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:49.992449000Z][com.docker.diagnose][I] (4db55475-4) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:49.994634000Z][com.docker.diagnose][W] (4db55475-4) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.176041ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:50.995807000Z][com.docker.diagnose][I] (4db55475-5) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:50.998085000Z][com.docker.diagnose][W] (4db55475-5) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.26975ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:51.999422000Z][com.docker.diagnose][I] (4db55475-6) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:52.001589000Z][com.docker.diagnose][W] (4db55475-6) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.164125ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:53.002692000Z][com.docker.diagnose][I] (4db55475-7) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:53.004767000Z][com.docker.diagnose][W] (4db55475-7) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.068667ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory
[2023-04-13T12:40:54.006103000Z][com.docker.diagnose][I] (4db55475-8) 3ad77a33-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-04-13T12:40:54.008382000Z][com.docker.diagnose][W] (4db55475-8) 3ad77a33-com.docker.diagnose C<-S NoResponse GET /ping (2.2805ms): Get "http://ipc/ping": dial unix lifecycle-server.sock: connect: no such file or directory

[FAIL] DD0015: are the binary symlinks installed? 2 errors occurred:
	* checking binaries in /usr/local/bin: looking for /usr/local/bin/docker: lstat /usr/local/bin/docker: permission denied
	* checking binaries in $HOME/.docker/bin: unable to find /Users/Rage/.docker/bin: stat /Users/Rage/.docker/bin: no such file or directory


[FAIL] DD0031: does the Docker API work? Cannot connect to the Docker daemon at unix://docker.raw.sock. Is the docker daemon running?
[FAIL] DD0013: is the $PATH ok? unable to find docker executable on PATH
[FAIL] DD0003: is the Docker CLI working? exec: "docker": executable file not found in $PATH
[PASS] DD0038: is the connection to Docker working?
[FAIL] DD0014: are the backend processes running? 3 errors occurred:
	* com.docker.vpnkit is not running
	* com.docker.driver.amd64-linux is not running
	* com.docker.virtualization is not running


[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[FAIL] DD0010: is the Docker API proxy responding? failed to ping Docker proxy API with error: Get "http://ipc/desktop-diagnostics/ping": read unix ->/Users/Rage/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)
[2023-04-13T12:40:55.747620000Z][com.docker.diagnose][I] ipc.NewClient: be681834-diagnose -> <HOME>/.docker/run/docker.sock Proxy
[2023-04-13T12:40:55.747962000Z][com.docker.diagnose][I] (2962e667) be681834-diagnose C->S Proxy GET /desktop-diagnostics/ping
[2023-04-13T12:40:56.749688000Z][com.docker.diagnose][W] (2962e667) be681834-diagnose C<-S NoResponse GET /desktop-diagnostics/ping (1.001707834s): Get "http://ipc/desktop-diagnostics/ping": read unix -><HOME>/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)
[2023-04-13T12:40:56.751581000Z][com.docker.diagnose][I] (2962e667-1) be681834-diagnose C->S Proxy GET /ping
[2023-04-13T12:40:57.754142000Z][com.docker.diagnose][W] (2962e667-1) be681834-diagnose C<-S NoResponse GET /ping (1.002534333s): Get "http://ipc/ping": read unix -><HOME>/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)
[2023-04-13T12:40:58.756890000Z][com.docker.diagnose][I] (2962e667-2) be681834-diagnose C->S Proxy GET /ping
[2023-04-13T12:40:59.759177000Z][com.docker.diagnose][W] (2962e667-2) be681834-diagnose C<-S NoResponse GET /ping (1.002291042s): Get "http://ipc/ping": read unix -><HOME>/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)
[2023-04-13T12:41:00.761361000Z][com.docker.diagnose][I] (2962e667-3) be681834-diagnose C->S Proxy GET /ping
[2023-04-13T12:41:01.763714000Z][com.docker.diagnose][W] (2962e667-3) be681834-diagnose C<-S NoResponse GET /ping (1.002354334s): Get "http://ipc/ping": read unix -><HOME>/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)
[2023-04-13T12:41:02.766400000Z][com.docker.diagnose][I] (2962e667-4) be681834-diagnose C->S Proxy GET /ping
[2023-04-13T12:41:03.769550000Z][com.docker.diagnose][W] (2962e667-4) be681834-diagnose C<-S NoResponse GET /ping (1.003080042s): Get "http://ipc/ping": read unix -><HOME>/.docker/run/docker.sock: i/o timeout (Client.Timeout exceeded while awaiting headers)

[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[WARN] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": dial unix diagnosticd.sock: connect: no such file or directory
[WARN] DD0004: is the Docker engine running? Get "http://ipc/docker": dial unix lifecycle-server.sock: connect: no such file or directory
[WARN] DD0015: are the binary symlinks installed? 2 errors occurred:
	* checking binaries in /usr/local/bin: looking for /usr/local/bin/docker: lstat /usr/local/bin/docker: permission denied
	* checking binaries in $HOME/.docker/bin: unable to find /Users/Rage/.docker/bin: stat /Users/Rage/.docker/bin: no such file or directory


[WARN] DD0031: does the Docker API work? Cannot connect to the Docker daemon at unix://docker.raw.sock. Is the docker daemon running?
[WARN] DD0032: do Docker networks overlap with host IPs? Cannot connect to the Docker daemon at unix://docker.raw.sock. Is the docker daemon running?

Steps to reproduce the behavior

Start Docker
Go to 'Settings -> Resources -> Advanced -> Disk Image location'
Change location to empty folder of external drive, which is not protected from the read/write
See the app hangs as on the screen I provided above

Apr 13 '23 13:04 dontturnaway

I found the similar issue here https://github.com/docker/for-mac/issues/6797, but there wasn't Diagnostics ID, so I'm afraid it would be closed. Don't want to unfocus this issue due a formal errors of templating by another author.

Apr 13 '23 13:04 dontturnaway

I have the same problem.

After reinstall version 4.16.0 (95345) and change disk image localtion works fine.

Then update to latest version 4.18.0 (104112)

Apr 18 '23 03:04 Abirdcfly

After reinstall version 4.16.0 (95345) and change disk image localtion works fine.

Then update to latest version 4.18.0 (104112)

So, to overcome this bug I need to downgrade first my 4.18 (104112) - > 4.16 and then upgrade again 4.16 - > 4.18? Right?

Apr 18 '23 06:04 dontturnaway

So, to overcome this bug I need to downgrade first my 4.18 (104112) - > 4.16 and then upgrade again 4.16 - > 4.18? Right?

Yes, by doing that, I successfully changed the disk image localtion to my external hard drive.

Apr 18 '23 06:04 Abirdcfly

Yes, by doing that, I successfully changed the disk image localtion to my external hard drive.

Are you able to uninstall Docker Desktop too using appropriate menu in program (Docker -> Troubleshoot -> Uninstall)? I was even unable to uninstall it to switch to the previous version you mention. At least, with using GUI. Screenshot 2023-04-19 at 11 50 33 Only /Applications/Docker.app/Contents/MacOS/uninstall works.

Apr 19 '23 09:04 dontturnaway

Yes, by doing that, I successfully changed the disk image localtion to my external hard drive.

Are you able to uninstall Docker Desktop too using appropriate menu in program (Docker -> Troubleshoot -> Uninstall)? I was even unable to uninstall it to switch to the previous version you mention. At least, with using GUI. Only /Applications/Docker.app/Contents/MacOS/uninstall works.

same.

Apr 19 '23 09:04 Abirdcfly

Same problem on latest version (4.19). Hangs indefinitely on changing the docker image directory.

May 02 '23 23:05 TomLBZ

I have the same problem.

After reinstall version 4.16.0 (95345) and change disk image localtion works fine.

Then update to latest version 4.18.0 (104112)

Same problem on latest version (4.19). Hangs indefinitely on changing the docker image directory.

After uninstall 4.19 and install 4.16.0, it can changes disk image location.

May 05 '23 09:05 khoa-io

Same issue.

May 15 '23 22:05 srp3rd

Any progress on this? same issue on Windows 10 Pro.

Docker Desktop v4.19.0

May 17 '23 06:05 MeowKim

Same here +1.

Thanks for the workaround with the downgraded versions, that worked for me. As a reference, check the Docker release notes to find where to download an older version: https://docs.docker.com/desktop/release-notes/#4163

May 18 '23 21:05 dzou

Same here on 4.20.0 (109717) at macOS Ventura, downgrading to 4.16.2 also worked

Jun 01 '23 14:06 GuilhermeMGBR

Another workaround that worked for me without downgrading and using a symlink to the external volume: https://www.howtogeek.com/devops/how-to-store-docker-images-and-containers-on-an-external-drive/

Jun 05 '23 06:06 jimmygchen

Still hangs on 4.21.1 (114176), but workaround with symlink works, but needs to point from data folder in first location to another folder in second location.

https://www.howtogeek.com/devops/how-to-store-docker-images-and-containers-on-an-external-drive/

Jul 19 '23 12:07 tomaszstachera

I did find a sort of halfway house to sorting this as I was experiencing the same issue, wanting to move my Docker.raw to an external SSD. This is on OSX Ventura. Just bear in mind with this approach you will lose all data in the Docker.raw as it will initialise a new one.

Here what I did:-

Left the disk image location as it was pointing to the Docker.raw on my host system (as I too had it just sit/freeze there when attempting to do it through the docker desktop UI)
Quit docker desktop entirely
Created a directory to host my Docker.raw file on the external SSD
Touch'ed a new/zero-byte Docker.raw file in the new directory on the external SSD
Renamed my existing Docker.raw on the host system out the way temporarily to make sure I could restore it back if this didn't work (you can then delete it later when you're happy this is working)
Symlinked the Docker.raw location where Docker desktop expects it to be to the new zero-byte Docker.raw on my external SSD
Startup docker desktop, it presumably realises the Docker.raw is corrupt and just generates a new one and away you go with it now writing to the Docker.raw on your SSD
Confirmed that there were no docker images in my install now
Ran a project build of something to confirm the Docker.raw on my external SSD was being written to

Just bear in mind this is just moving Docker.raw and not any of the log files etc. I've not bothered moving the rest although I'm guessing symlinking the directory itself further up the path would likely work, but I only really care about moving the main docker disk image file.

Why didn't I copy the Docker.raw to the SSD to retain my existing docker images/volumes etc.?

Well you could and maybe that would work, but it appears as though the file has a "virtual allocation" of what you set it to (128GB in my case) and doing a copy of that will copy the entire 128GB file so you'll immediately waste a load of space that's not actually been written to. So for me 30GB used out of 128GB would have wasted near 100GB of space on the SSD.

I'm sure there's maybe a way of copying while keeping the virtual allocation intact through dd or something, but didn't bother to look too deeply into this side of things.

I can't vouch for the stability of this hack (e.g., not checked/ran into what happens if the disk isn't attached and docker is running or it started up), but appears to be working. I guess you'd probably risk maybe Docker.raw file corruption which it can't recover from and you have to re-initialise a fresh Docker.raw.

I hope this helps someone.

Jul 20 '23 12:07 sparkcodeuk

I met the same question. I tried some ways but it failed. Then when I changed my "virtual disk limit" and "Disk image location", and clicked the"Apply & restart" button both, it worked. I hope this can help. My docker desktop version is 24.0.2

Jul 21 '23 02:07 Jingogogo

I'm encountering this issue on v4.21.1 (114176)

Aug 01 '23 11:08 gyrex

Me too

Aug 02 '23 14:08 LookKit

same issue with v4.19.0. It costs 60GB, too much to sustain in 256GB system disk. It would be nice to get solved soon. Hopefully 4.21.2?

Aug 02 '23 15:08 gaord

Same issue for me with 4.22.0 (117440) on Ventura 13.5 (22G74) using the "Virtualization Framework" option (if that matters). It also seems to take it sometimes without hanging but then a restart of the Docker Desktop app causes it to revert back to the original location and never use the new one.

Update The below is a work-around that appears to work for me at least:

Quit the Docker Desktop application completely.
Make sure no remaining Docker processes are running: ps -efww | grep Docker
Create a new directory on your external SSD: mkdir -vp /Volumes/<Name>/DockerDesktop/vms
Goto the default data storage directory: cd ~Library/Containers/com.docker.docker/Data
Remove the existing vms directory: rm -rf vms
Replace with a symlink to the above new directory: ln -s /Volumes/<Name>/DockerDesktop/vms .
Start the Docker Desktop application.

Aug 10 '23 14:08 robert914

Had the same problem on Ventura 13.4.1 (Intel i5) and Docker 4.21.1

Got it working after doing a clean/purge and reset to factory defaults.

Aug 24 '23 16:08 moroboro

Anyone got a solution to this yet? (On mac docker desktop). I have about 15gigs of space left on my main drive and a nvme high speed ext hard drive (Takes about 15 seconds to move 30gigs of ram onto it, its fast) to move it to, but it shits the bed everytime I try.

Aug 31 '23 04:08 shayneoneill

Anyone got a solution to this yet? (On mac docker desktop). I have about 15gigs of space left on my main drive and a nvme high speed ext hard drive (Takes about 15 seconds to move 30gigs of ram onto it, its fast) to move it to, but it shits the bed everytime I try.

Uninstal your current docker installation then install v4.16.xx instead , then move your docker image to external ssd trough setting. Verify docker installation by running some docker sample, and if it success then you can update docker desktop to latest version. This way works on my mac m1 air with 512 GB external NVME SSD

Aug 31 '23 06:08 wahyudotdev

I couldn't change Disk image location even if I installed v4.16.2 Mac with Apple chip.

Uninstalle v4.21.x
Installed v4.16.2 (I tried also v4.16.0, v.4.15.0, but same)
Start Docker Desktop
Apply change of Disk image location. But actually the circle of progress bar constantly spins and has never stoped
Force quit Docker DEsktop
Restart Docker Desktop. It make DockerDesktop/Docker.raw file in designated location but Docker Desktop starting... doesn't finish...

Sep 04 '23 08:09 tegnike

Above 4 situation スクリーンショット 2023-09-04 10 45 15

Sep 04 '23 08:09 tegnike

Confirming the same issue on the following system:

latest and fresh macOS 13.5.2 (22G91)
latest and fresh Docker Desktop 4.22.1 (118664)
Intel based MacMini (3 GHz 6-Core Intel Core i5).

The "Disk Image Location" setting doesn't work. The new location folder is always empty. Also if I change the "Virtual Disk Limit" for example, I see that the size of the default folder (/Users/$USER/Library/Containers/com.docker.docker) is changed. Restarting OS or Docker engine doesn't have any effect. Deleting the default folder also leads to it's recreation on the default location and not on a new path.

Almost zero documentation, big tech company and such a result... Cannot believe.

Sep 10 '23 12:09 korobochkin

Downgrading to 4.16.2 did not work for me. Docker still locks up after applying the disk image location change.

I'm at a loss. Will I have to buy a new computer just to use Docker? I'm out of disk space. What alternatives are there?

Edit: after restarting docker 3x, magically it's working.

Edit 2: It seems like Docker (maybe MacOS Monterey in general) has issues with ExFAT filesystems, switching to APFS resolved issues such as incorrect no space left on device

Sep 20 '23 02:09 jackofsometrade

Almost zero documentation, big tech company and such a result... Cannot believe.

Sep 22 '23 02:09 mushroomlb

Downgrading to 4.16.2 did not work for me. Docker still locks up after applying the disk image location change.

I'm at a loss. Will I have to buy a new computer just to use Docker? I'm out of disk space. What alternatives are there?

Edit: after restarting docker 3x, magically it's working.

Edit 2: It seems like Docker (maybe MacOS Monterey in general) has issues with ExFAT filesystems, switching to APFS resolved issues such as incorrect no space left on device

I've got this working on my macbook m1 using SSD