docker
Syncronized file share broken since v4.34.2
Description
Something in the update from v4.32 to v4.34.2 completely broke all file shares on Mac OS.
Reproduce
- Enable file share
- Change a file
- Try to access the updated file in the share
Result: it never changes in the share even though the "status" says it is up-to-date.
Expected behavior
The file should be updated in the share.
docker version
Client:
Version: 27.2.0
API version: 1.47
Go version: go1.21.13
Git commit: 3ab4256
Built: Tue Aug 27 14:14:45 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.34.2 (167172)
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:41 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 27.2.0
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2-desktop.1
Path: /Users/nikbr/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.2-desktop.2
Path: /Users/nikbr/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.34
Path: /Users/nikbr/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.15
Path: /Users/nikbr/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/nikbr/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.25
Path: /Users/nikbr/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/nikbr/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.3.0
Path: /Users/nikbr/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/nikbr/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.13.0
Path: /Users/nikbr/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/nikbr/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/nikbr/.docker/cli-plugins/docker-scan: no such file or directory
Server:
Containers: 28
Running: 12
Paused: 0
Stopped: 16
Images: 72
Server Version: 27.2.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.4-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 9
Total Memory: 23.44GiB
Name: docker-desktop
ID: 4bacd5ea-52d7-4544-a700-b29136ca8a38
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/nikbr/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
39263439-8ED3-425A-A6C1-A4B1DD929F05/20240916085406
Additional Info
No response
Hey @NiklasBr, I'm unable to reproduce this issue on my machine, but maybe I can get the information needed from your diagnostics bundle (EDIT: I didn't realize you had already uploaded one, apologies).
Does the Synchronize File Shares pane indicate any sort of errors, problems, or conflicts for the file sync?
Finally, were these synchronized file shares that existed prior to 4.34.2? Or does the problem occur for both existing synchronized file shares and new ones?
Looking at the diagnostics, I don't see any errors that would prevent the changes from propagating. If you exit and restart Docker Desktop, do the changes propagate after that?
Does the Synchronize File Shares pane indicate any sort of errors, problems, or conflicts for the file sync?
It indicates no such things, it's all "fine" according to the settings section.
Finally, were these synchronized file shares that existed prior to 4.34.2?
These existed for a while prior to 4.34.2, maybe since 4.29 or thereabout.
If you exit and restart Docker Desktop, do the changes propagate after that?
Yes, they propagate once then it stops. No further changes propagate into the running container. If I then restart Docker the changes propagate, but then no further changes propagate until I restart it again.
Any ideas @xenoscopic? I have downgraded to v4.32.0 and it restores working file shares without needing to do anything. But I can't run outdated insecure software for too long.
@NiklasBr Unfortunately not yet. I was testing this morning with DD 4.29, creating shares and then upgrading to 4.34.2, but all changes continue to propagate, including after the first. I've also looked through more of the logs in the diagnostic bundle you sent but haven't seen anything yet. I will get some time allocated during our next sprint (starting today) to delve further and hopefully get this sorted out. Is there any chance we could do a short (20-30 minute) screen share session early next week to walk through debugging on your machine? This may be related to docker/for-mac#7281, so perhaps we can follow up on that one as well.
Sure, send me an invite for Monday after 13 o'clock or Wednesday after 10 (CET), you have my email already.
This is still an issue in 4.37
See also
https://github.com/docker/for-mac/issues/7416 https://github.com/docker/for-mac/issues/7438
Still an issue in 4.38
See also
https://github.com/docker/for-mac/issues/7416 https://github.com/docker/for-mac/issues/7438
Diagnostics ID: D3BE585F-6A8B-441B-BDE0-33427C4EF3C3/20250205081407
Even though we have this .syncignore in the root of the project (next to the docker-compose.yaml file)
.git/
.idea/
pimcore/var/cache/dev/
We are seeing errors because of the files in pimcore/var/cache/dev/ directory:
Host: var/cache/dev/translations/catalogue.pt_br.HRGLoVq.php
unable to create file: unable to relocate staged file: case conflict or file already exists
This worked for a while but now doesn't.
Hey @bnomis, it looks like https://github.com/docker/for-mac/issues/7416 and https://github.com/docker/for-mac/issues/7438 are referring to virtual filesharing (either Virtiofs or gRPC-FUSE, depending on the configuration) and are unrelated to synchronized file shares.
@NiklasBr the .syncignore needs to be placed in the root of the synchronization point (in this case the pimcore directory) rather than the project root. The sync won't scan above it. In that case, the ignore pattern would need to be updated to var/cache/dev. That should resolve the case conflicts. Note that the ignored files won't be removed on either side, they just won't be synchronized anymore.
We are using the entire workspace as a bind mount, #7281, not subdirectories.
@NiklasBr in the diagnostic you sent, it looks like the problems are occurring on a synchronized file share located at <HOME>/w.../g.../pimcore (paths truncated for privacy). In that case, the .syncignore should be inside pimcore and shouldn't have a pimcore prefix for ignored paths.
I have not created any share at "~/workspace/g.../pimcore", after a previous discussion about problems with these File Shares I removed all shares and only added one: ~/workspace/.
However, now when I check the settings in Docker Desktop, I see a lot of shares I know I have not added. Some with only one or a few files, some which looks a lot like mounted volumes. Is Docker suddenly creating these automatically @xenoscopic?
@NiklasBr There was a default setting that was temporarily changed a while back (I think circa July 2024 but possibly later) which temporarily enabled the Compose + Synchronized File Shares integration for users who had not opted into it. While it was active, any Compose project would have automatically created synchronized file shares, which would have persisted even after the default was reverted. You can check if you're still opted into this by going to Settings > Features in development > Experimental features. You'll want to look for the "Manage Synchronized file shares with Compose" feature and ensure that it is unchecked. If it is, and if you delete the extra shares, then they shouldn't return. I apologize if that change of default caused the trouble.
With the "extra" shares, when a container is started, the bind mount will be matched to the "most specific" share (i.e. the share with the longest path that can satisfy the bind mount). That might be why you were seeing inconsistent behavior with the .syncignore - the more specific shares were being used and weren't being affected by the .syncignore in the parent directory.
That's probably it 👍 Manage Synchronized file shares with Compose was checked.
Now I need to figure out how to delete these shares. The button is greyed out with no tooltip…
Unfortunately you won't be able to delete them while there are containers using them since it would remove the filesystem location that they're bind-mounted to; the containers would have to be removed (not just stopped) to remove the shares.
What more than stopping the containers and executing docker compose down -v do I need to do?
That should be it. If the Delete button is still greyed out, then there should be a list of containers next to it indicating which containers are still using the share.
I have disabled Manage Synchronized file shares with Compose and removed shared storage, I have also updated to Mac OS 15.3.1 (and restarted the computer), yet the shares are re-created and problems persists:
@NiklasBr I don't have any good suggestions at the moment. I am investigating a few things internally with regards to why the shares are being automatically recreated and I will get back to you next week. I will also take a look at your new diagnostics bundle, but something looks to be significantly broken with that file share. Out of curiosity, have you fried a factory reset of Docker Desktop? I'm not claiming that will help, but knowing if it doesn't help would be useful.
I did factory reset my Docker installation some weeks ago where I completely wiped it from the system (including data in the Library) directory and downloaded a fresh copy (#7527).
In this latest report yesterday I had manually created a single share for the individual project where the share root is the same as the project root where the .syncignore file resides.