for-mac Permissions issue with VirtioFS

[x] I have tried with the latest version of Docker Desktop
[x] I have tried disabling enabled experimental features
[x] I have uploaded Diagnostics
Diagnostics ID:

Summary / Steps to reproduce

I started a gitlab container with docker compose. This container has a volume:

<some-local-folder>:/var/opt/gitlab:cached

what I did in the running container (note the commands access a sub-folder of the volume):

chmod 2770 /var/opt/gitlab/git-data/repositories
stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$'

Expected behavior

I would expect the permissions of this folder are as set by the chmod command before:

Therefore the output of the stat command should be:

Actual behavior

output of the stat command with setting "gRPC FUSE" of Docker Desktop (this is ok):

output of the stat command with setting "VirtioFS" of Docker Desktop (this is wrong!):

Btw.: gitlab fails on startup because of this issue.

Information

The problem should be reproducible with any other container. The problem is since I switched my settings to VirtioFS. With setting gRPC FUSE it works.

macOS Version: 13.0.1
Intel chip or Apple chip: Intel
Docker Desktop Version: 4.15.0 (93002)

Output of `/Applications/Docker.app/Contents/MacOS/com.docker.diagnose check`

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0012: is the VM networking working?
[SKIP] DD0030: is the image access management authorized?
[FAIL] DD0019: is the com.docker.vmnetd process responding? failed to ping vmnetd with error: failed to connect to /var/run/com.docker.vmnetd.sock: is vmnetd running?: dial unix /var/run/com.docker.vmnetd.sock: connect: no such file or directory
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?

Please investigate the following 1 issue:

1 : The test: is the com.docker.vmnetd process responding?
    Failed with: failed to ping vmnetd with error: failed to connect to /var/run/com.docker.vmnetd.sock: is vmnetd running?: dial unix /var/run/com.docker.vmnetd.sock: connect: no such file or directory

The com.docker.vmnetd process is needed to create symlinks for CLIs in your path.

Steps to reproduce the behavior

See above.

Dec 04 '22 19:12 abentele

Still testing this, but might be related to what the OP is experiencing: since updating to 4.15, when I git pull the files are set to 755 instead of 644. My umask seems to be defaulting to what I expect (0022). I don't seem to have any problems with the other virtualization modes available.

Dec 05 '22 09:12 niccolomineo

Thanks for reporting; I just hit this issue myself; I'm discussing with the team working on VirtioFS, and they're looking into it.

Dec 05 '22 13:12 thaJeztah

Same here. For some reason all files inside my devcontainer are now marked as executable after a git pull.

Dec 05 '22 16:12 fasmat

Same here, rolling back to 4.14.1 solves the problem

Dec 06 '22 13:12 bramru

Short update; I see a patch is being worked on to fix this (but not finished yet); from the description, it looks like there may be a bug on Apple's side, but we're looking at a workaround until that's fixed.

Dec 09 '22 23:12 thaJeztah

I can also confirm this issue. It's really creating havoc while working in a VSCode devcontainer. Files randomly reset their permissions from 0644 to 0755 in the container. Then if try to run a git status in the host OS (macOS Monterey) git thinks ALL of the files have changed (or a ton of them) and a git diff indicates that the permissions have changed (from 0755 to 0644). This might be impacting my ability to rebase as well but not sure.

Dec 12 '22 03:12 markeissler

Switched back to "gRPC FUSE" for now. Seems like a good enough work around until a fix is in place.

Dec 18 '22 01:12 markeissler

Hey there, this build no longer sets the executable bit on your volume files : amd64: https://desktop-stage.docker.com/mac/main/amd64/94201/Docker.dmg arm64: https://desktop-stage.docker.com/mac/main/arm64/94201/Docker.dmg Let us know if any issue occurs, Thanks!

Dec 19 '22 14:12 fredericdalleau

@abentele note that this executable bit issue is different for the issue you're facing. Investigation continues.

Dec 19 '22 16:12 fredericdalleau

@fredericdalleau my team experienced the same problem with permissions in dev container on 4.15. The build that you've linked resolves the problem for us 👍

Is this likely to be released soon?

Dec 20 '22 12:12 oinopion

@oinopion thanks for feedback, it should land in our next release early next year

Dec 20 '22 13:12 fredericdalleau

@fredericdalleau Thanks a lot for this Docker build!

Dec 28 '22 12:12 umbrellait-danil-rodin

4.16 preview works like a charm on my system, thank you for that. 👍

Dec 28 '22 14:12 niccolomineo

Closing this issue because a fix has been released in Docker Desktop 4.16. See the release notes for more details.

Jan 13 '23 17:01 mat007

Sadly we are still having issues with 4.16.1 over at https://github.com/lando/lando which is unfortunate because our users definitely crave being able to use VirtioFS.

4.16.1 produces breaking behavior because it does not mount our boot up entrypoint scripts with the same permissions they have on the host.

This is what the perms look like for our scripts. Is this intentional?

total 544
     0 drwxr-xr-x  134 501      dialout       4288 Nov 18 19:53 .
     4 drwxr-xr-x    1 root     root          4096 Jan 13 18:15 ..
     8 -rwxr-xr-x    1 501      dialout       6148 Oct 26  2021 .DS_Store
     4 -------r--    1 501      dialout       1248 May  6  2020 999-proxy-certs.sh
     4 -------r--    1 501      dialout        811 Jan 13 18:15 acquia-clone.sh
     4 -------r--    1 501      dialout        242 Jan 13 18:15 acquia-config-symlink.sh
     4 -------r--    1 501      dialout        322 Jan 13 18:15 acquia-generate-key.sh
     4 -------r--    1 501      dialout       1188 Mar 17  2021 acquia-get-remote-url.sh
     4 -------r--    1 501      dialout       3152 Jan 13 18:15 acquia-pull.sh
     4 -------r--    1 501      dialout       2087 Jan 13 18:15 acquia-push.sh
     4 -------r--    1 501      dialout        509 Mar 13  2021 acquia-settings.inc
     8 -------r--    1 501      dialout       4319 Jan 13 18:15 add-cert.sh
     4 -------r--    1 501      dialout       2920 Sep 15  2021 agent.sh
     4 -------r--    1 501      dialout       1456 Jan 13 18:15 auth.sh
...

Jan 13 '23 18:01 pirog

Can confirm issues with 4.16.1 as well.

We're building protobuf files on a host-mounted volume which results in temp files with no permissions (which ultimately leads to a proto build failure):

---------- 1 jalaziz staff     0 Jan 15 02:10 sednOBSOT

Jan 15 '23 10:01 jalaziz

@pirog, can you share what command you used to get this result?

Jan 17 '23 07:01 fredericdalleau

@fredericdalleau sadly i cannot!

I reset Docker Desktop to factory defaults and rebooted my computer and now it seems to work as expected. At least for me. Does that make any sense to you?

Jan 17 '23 17:01 pirog

You may need to reenable virtiofs as it not default. If it happens again, you can try on the host : xattr -l <filename> and see there is any output. If so you can clear it with xattr -c <filename>, and set the permissions again. (and share the output here)

Jan 17 '23 18:01 fredericdalleau

@fredericdalleau good to know.

We have some users reporting the same initial no-perms behavior so i'm going to walk through it with a few of them and hopefully i'll be able to track things down a bit better for you all.

Jan 17 '23 19:01 pirog

I've got this issue too, so I've to downgrade to 3.15 which works, even with VirtioFS enabled. On 3.16.x it breaks when using VirtioFS.

Using Apple Silicon.

Jan 27 '23 16:01 renkosteenbeek

On 4.16.2 and also seeing this issue; Mac pro with Intel chip. It's amazing that this feature isn't still in the "Features in development" section! Permissions is a massively important thing to get right.

Lmk if you'd like a set of repro steps (although fair warning; it's not a simple setup)

Feb 05 '23 17:02 xEtherealx

@xEtherealx Thanks for reporting, can you share share your reproduction steps? For a complex setup, it would be very kind to prepare a git repo with a script or docker compose file.

Feb 07 '23 14:02 fredericdalleau

@fredericdalleau it's not a super complex setup actually, just a bit time consuming because the build process is lengthy. Here are the repro steps:

Clone this fork of batocera.linux, with the osxsupport branch which adds the ability to kick off build on OSX https://github.com/xEtherealx/batocera.linux/tree/osxsupport
Make sure that virtiofs is enable in docker desktop
in the root directory, run make build-docker-image and then once the docker image is built, make V=1 bcm2837-build

That's it, the build may take a while but results in weird permissions issues reliably for me

Feb 07 '23 19:02 xEtherealx

@xEtherealx An error occurs : No rule to make target batocera-bcm2837_defconfig. It also occurs on Linux host on master branch.

Feb 08 '23 10:02 fredericdalleau

Hello,

I also have this behaviour. I recently switched with my smarthome from Ubuntu (with Docker) to Mac OS with Docker. I am having issues with VirtioFS with GitLab and with Nodered.

I know how to reproduce this problem.

Use this docker-compose file

version: "2.4"
services:
        nodered:
                image: nodered/node-red:latest
                ports: 
                        - 1881:1880
                volumes:
                        - /Users/username/Docker/Volumes/nodered_test:/data
                restart: unless-stopped
        %

Then access nodered via browser http://localhost:1881 Click on the top right, then settings -> palette -> install. Search for node-red-dashboard and try to install it. It will abort with an error message. As soon, as you switch back to gRPC FUSE the installation will work fine. Also it's not for all nodered modules, but definitely for this one

-----------------------------------------------------------
2023-02-08T10:32:16.552Z Install : node-red-dashboard 3.3.1

2023-02-08T10:32:16.567Z npm install --no-audit --no-update-notifier --no-fund --save --save-prefix=~ --production --engine-strict [email protected]
2023-02-08T10:32:16.868Z [err] npm
2023-02-08T10:32:16.868Z [err]  WARN config production Use `--omit=dev` instead.
2023-02-08T10:32:17.788Z [err] npm WARN tarball tarball data for ms@https://registry.npmjs.org/ms/-/ms-2.1.2.tgz (sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.790Z [err] npm WARN
2023-02-08T10:32:17.790Z [err]  tarball tarball data for ms@https://registry.npmjs.org/ms/-/ms-2.1.2.tgz (sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.848Z [err] npm WARN tarball tarball data for debug@https://registry.npmjs.org/debug/-/debug-4.3.4.tgz (sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.852Z [err] npm WARN tarball tarball data for debug@https://registry.npmjs.org/debug/-/debug-4.3.4.tgz (sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.888Z [err] npm WARN tarball tarball data for ms@https://registry.npmjs.org/ms/-/ms-2.1.2.tgz (sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.892Z [err] npm WARN tarball tarball data for ms@https://registry.npmjs.org/ms/-/ms-2.1.2.tgz (sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.938Z [err] npm WARN tarball tarball data for debug@https://registry.npmjs.org/debug/-/debug-4.3.4.tgz (sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==) seems to be corrupted. Trying again.
2023-02-08T10:32:17.939Z [err] npm WARN 
2023-02-08T10:32:17.939Z [err] tarball tarball data for debug@https://registry.npmjs.org/debug/-/debug-4.3.4.tgz (sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==) seems to be corrupted. Trying again.
2023-02-08T10:32:18.768Z [err] npm ERR! code ENOENT
2023-02-08T10:32:18.768Z [err] npm ERR! syscall lstat
2023-02-08T10:32:18.768Z [err] npm ERR! path /data/.npm/_cacache/content-v2/sha512/3d/15/851ee494dde0ed4093ef9cd63b25c91eb758f4b793ae3ac1733cfcec7a40f9d9997ca947c520f122b305ea22f1d61951ce817fbb1bfbc234d85e870c5f91
2023-02-08T10:32:18.769Z [err] npm ERR! errno -2
2023-02-08T10:32:18.769Z [err] npm ERR! enoent ENOENT: no such file or directory, lstat '/data/.npm/_cacache/content-v2/sha512/3d/15/851ee494dde0ed4093ef9cd63b25c91eb758f4b793ae3ac1733cfcec7a40f9d9997ca947c520f122b305ea22f1d61951ce817fbb1bfbc234d85e870c5f91'
2023-02-08T10:32:18.770Z [err] npm ERR! enoent This is related to npm not being able to find a file.
2023-02-08T10:32:18.770Z [err] npm ERR! enoent 
2023-02-08T10:32:18.773Z [err] 
2023-02-08T10:32:18.774Z [err] npm ERR! A complete log of this run can be found in:
2023-02-08T10:32:18.774Z [err] npm ERR!     /data/.npm/_logs/2023-02-08T10_32_16_843Z-debug-0.log
2023-02-08T10:32:18.804Z rc=254

Feb 08 '23 10:02 thomas-schwieters

@thomas-schwieters could it relate to a long path? Could you try with a shorter volume path ?

                        - /Users/username/nodest:/data

Feb 09 '23 13:02 fredericdalleau

I just tried it with this path:

                volumes:
                        - /Users/thomasschwieters/nodered:/data

Same error:

thomasschwieters-nodered-1  | Your flow credentials file is encrypted using a system-generated key.
thomasschwieters-nodered-1  | 
thomasschwieters-nodered-1  | If the system-generated key is lost for any reason, your credentials
thomasschwieters-nodered-1  | file will not be recoverable, you will have to delete it and re-enter
thomasschwieters-nodered-1  | your credentials.
thomasschwieters-nodered-1  | 
thomasschwieters-nodered-1  | You should set your own key using the 'credentialSecret' option in
thomasschwieters-nodered-1  | your settings file. Node-RED will then re-encrypt your credentials
thomasschwieters-nodered-1  | file using your chosen key the next time you deploy a change.
thomasschwieters-nodered-1  | ---------------------------------------------------------------------
thomasschwieters-nodered-1  | 
thomasschwieters-nodered-1  | 9 Feb 14:08:36 - [info] Server now running at http://127.0.0.1:1880/
thomasschwieters-nodered-1  | 9 Feb 14:08:36 - [warn] Encrypted credentials not found
thomasschwieters-nodered-1  | 9 Feb 14:08:36 - [info] Starting flows
thomasschwieters-nodered-1  | 9 Feb 14:08:36 - [info] Started flows
thomasschwieters-nodered-1  | 9 Feb 14:09:18 - [info] Installing module: node-red-dashboard, version: 3.3.1
thomasschwieters-nodered-1  | 9 Feb 14:09:20 - [warn] Installation of module node-red-dashboard failed:
thomasschwieters-nodered-1  | 9 Feb 14:09:20 - [warn] ------------------------------------------
thomasschwieters-nodered-1  | 9 Feb 14:09:20 - [warn] npm WARN config production Use `--omit=dev` instead.
thomasschwieters-nodered-1  | npm ERR! code ENOENT
thomasschwieters-nodered-1  | npm ERR! syscall rename
thomasschwieters-nodered-1  | npm ERR! path /data/.npm/_cacache/tmp/07005d37
thomasschwieters-nodered-1  | npm ERR! dest /data/.npm/_cacache/content-v2/sha512/49/df/a5a5e22482b4bae15832f900fe23bcbae5ed60665f85dd2f0c10ef05b9e47df28b0c7a479de5651e6a60052838ff4db5291bf1eddbec925a611953b34d91
thomasschwieters-nodered-1  | npm ERR! errno ENOENT
thomasschwieters-nodered-1  | npm ERR! enoent Invalid response body while trying to fetch https://registry.npmjs.org/debug: ENOENT: no such file or directory, rename '/data/.npm/_cacache/tmp/07005d37' -> '/data/.npm/_cacache/content-v2/sha512/49/df/a5a5e22482b4bae15832f900fe23bcbae5ed60665f85dd2f0c10ef05b9e47df28b0c7a479de5651e6a60052838ff4db5291bf1eddbec925a611953b34d91'
thomasschwieters-nodered-1  | npm ERR! enoent This is related to npm not being able to find a file.
thomasschwieters-nodered-1  | npm ERR! enoent 
thomasschwieters-nodered-1  | 
thomasschwieters-nodered-1  | npm ERR! A complete log of this run can be found in:
thomasschwieters-nodered-1  | npm ERR!     /data/.npm/_logs/2023-02-09T14_09_18_402Z-debug-0.log
thomasschwieters-nodered-1  | 
thomasschwieters-nodered-1  | 9 Feb 14:09:20 - [warn] ------------------------------------------
thomasschwieters-nodered-1  | Error: Install failed
thomasschwieters-nodered-1  |     at /usr/src/node-red/node_modules/@node-red/registry/lib/installer.js:285:25
thomasschwieters-nodered-1  |     at processTicksAndRejections (node:internal/process/task_queues:96:5)
thomasschwieters-nodered-1  | 9 Feb 14:09:20 - [error] Error: Install failed
thomasschwieters-nodered-1  | 9 Feb 14:10:03 - [info] Stopping flows
thomasschwieters-nodered-1  | 9 Feb 14:10:03 - [info] Stopped flows

Feb 09 '23 14:02 thomas-schwieters

@thomas-schwieters do you have a curl request that could be used after starting the container that triggers the error?

Feb 09 '23 14:02 fredericdalleau

No, I don't have the curl command, but you can execute this command inside the container:

npm install --no-audit --no-update-notifier --no-fund --save --save-prefix=~ --production --engine-strict [email protected]

Feb 09 '23 14:02 thomas-schwieters

for-mac for-mac copied to clipboard

Permissions issue with VirtioFS

Summary / Steps to reproduce

Expected behavior

Actual behavior

Information

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Steps to reproduce the behavior

for-mac
for-mac copied to clipboard

Output of `/Applications/Docker.app/Contents/MacOS/com.docker.diagnose check`