docker
VirtioFS: File size in bind mounted volume does not match host
Description
This might have some similarity to issue #7494.
The development setup where I work uses a project volume shared by multiple containers. The volume is also bind mounted to a directory on the host using mount options:
docker volume create \
--driver local \
--opt type="none" \
--opt o="bind" \
--opt device="$HOST_DIRECTORY" \
"$VOLUME_NAME"
This has worked fine in Docker Desktop on Mac, using VirtioFS (and the "Apple Virtualization Framework"), with Docker Desktop up until version 4.32. The volume is synced any host file changes due to being bind mounted.
When upgrading to version 4.33 we started having issues with files being "truncated" when accessed by containers when there are changes to the file on the host. The contents are updated on the container, but do not fully match the host.
For example, a file with contents: ABCDEFG on the host, after adding some characters to it, (e.g. making the contents 123ABCDEFG), when accessed by the container through the volume, will be truncated (e.g. 123ABCD), with the amount of characters added being absent at the end of the file.
By checking with ls on both the container and host, I was able to determine the file size is not updated correctly on the volume, but the contents are.
Looking at this comment on the issue I referenced above, I suspect this might also be related to VirtioFS cache invalidation not being triggered. If I run this sequence of commands from inside a container mounting the volume:
cat file.txt
# Get corrupted/truncated contents.
ls
cat file.txt
# File content is no longer truncated and matches the host.
Anyway, I was able to reproduce it consistently with the following script:
Reproduce
VOLUME_NAME="testing-volume"
DIRECTORY="directory"
FILE="test.txt"
docker volume create --driver local --opt type="none" --opt o="bind" --opt device="$(pwd)" "$VOLUME_NAME"
mkdir $DIRECTORY
echo "123456789" > $DIRECTORY/$FILE
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "123456789"
echo "ABC123456789" > $DIRECTORY/$FILE
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "ABC1234567%" (% representing no end of line character on my terminal, as the contents are truncated)
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine ls -la $DIRECTORY/$FILE
# Output for the file is: "-rw-r--r-- 1 root root 10 Dec 10 14:03 directory/test.txt", notice 10 characters when it's now 13
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine ls -la $DIRECTORY
# Output for the file is: "-rw-r--r-- 1 root root 13 Dec 10 14:04 test.txt", size is updated
docker run -it -w /var/workdir --mount source="$VOLUME_NAME",target=/var/workdir --rm alpine cat $DIRECTORY/$FILE
# Output is "ABC123456789", the correct contents on the host
Expected behavior
The file in the bind mounted volume should match the file on the host on every change.
docker version
Client:
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:38:18 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.36.0 (175267)
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:19 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 27.3.1
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Ask Gordon - Docker Agent (Docker Inc.)
Version: v0.1.0
Path: /Users/lorenzo/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.18.0-desktop.2
Path: /Users/lorenzo/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.30.3-desktop.1
Path: /Users/lorenzo/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.37
Path: /Users/lorenzo/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.15
Path: /Users/lorenzo/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/lorenzo/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: /Users/lorenzo/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/lorenzo/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/lorenzo/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/lorenzo/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.15.0
Path: /Users/lorenzo/.docker/cli-plugins/docker-scout
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 15
Server Version: 27.3.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 7.653GiB
Name: docker-desktop
ID: 1e75072f-7d8f-47c3-917a-43dc08d31755
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/lorenzo/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
Diagnostics ID
4EB0271E-BBC1-4A04-87BB-E7F66581011B/20241210141531
Additional Info
This behavior happened only with VirtioFS, the same setup when run with the "osxfs (Legacy)" file sharing implementation had no issues.
This might be of note: when I attempted running the same reproduction script above with the "Docker VMM (Beta)" Virtual Machine Manager, the file is never updated on the volume after changes on the host, it was not corrupted/truncated, it is just never updated.
Hello,
I experience the same issue using Docker Desktop 4.37.0 (178034)
I confirm that switching from VirtioFS to osxfs (Legacy) solved the issue regarding file sharing
On top of what @lorenzodalaqua said, I encountered a strange issue where files where not truncated, but changes in files done on my IDE was not directly reflected in the docker container (I have a flask app with gunicorn that should reload on file changes, but no reload was happening after modifying the code). Stranger, I discovered that, if I am in the container and I do a ls in the folder in which code changed, reload was happening at this moment. It might indeed be related to VirtioFS cache invalidation not being triggered
Checking in with the same problem. MacOS Sequoia 15.1.1, 2021 M1 Max. I suspected this might be something related to inotify and/or atomic save; the issue is persistent across Sublime Text and VSCode, but intermittent when files are edited with vim.
🤷
docker info
Client:
Version: 27.3.1
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Ask Gordon - Docker Agent (Docker Inc.)
Version: v0.1.0
Path: /Users/wyatt/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.18.0-desktop.2
Path: /Users/wyatt/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.30.3-desktop.1
Path: /Users/wyatt/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.37
Path: /Users/wyatt/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.15
Path: /Users/wyatt/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/wyatt/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: /Users/wyatt/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: /Users/wyatt/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/wyatt/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/wyatt/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.15.0
Path: /Users/wyatt/.docker/cli-plugins/docker-scout
Server:
Containers: 29
Running: 20
Paused: 0
Stopped: 9
Images: 24
Server Version: 27.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 16.07GiB
Name: docker-desktop
ID: a42c5b6e-dfc2-4245-9d34-b5b43ff43dd7
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/wyatt/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
docker version
Client:
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:38:18 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.36.0 (175267)
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:19 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
@bsousaa is there somewhere that we can use to track the progress of resolving this issue?
I haven't tried it yet, but 4.38.0 was released yesterday and has this fix listed:
- Fixed a bug in Docker VMM where bind-mounts from non-root volumes would weren't working as expected.
The problem seems to be improved, but we are still experiencing the issue to a limited degree. We will provide more specifics once we have had a chance to investigate a way to reproduce the problem consistently.
Same issue here (4.39.0). Particularly when switching git branches that dramatically changes a lot of files at once - almost never works properly - requiring restarting of the entire container for the state to be correct once more.
I have performed a full OS re-install of Mac OS 15.3.1 and installed Docker 4.39.0, and the problem still exists exactly as described in Lorenzo's original post. The script he provided recreates the problem consistently.
I must have had a different problem, similar symptoms. I just killed fseventsd (finally remembered I'd seen this before) and everything started working properly again. Not suggesting this is a solution for all - if you've already rebooted and that didn't help I doubt this would.
@KatTomrushka can you please give us any update on the status of this? Thank you.
We switched our projects to Bind Mounts in the hope that it would work around this problem, but we are seeing this happen even with Bind Mounts now. 😩
Still happening on Docker 4.40.0. It'd be awesome if ya'll would give us some feedback on fixing this. Anything would be nice...
Hello, apologies for the delay - we have a fix ETA 4.42 currently scheduled 29 May.
