chown error on bind mount when trying to launch postgres via docker compose

[valouille]

Rancher Desktop Version

0.7.1

Rancher Desktop K8s Version

1.22.5

What operating system are you using?

macOS

Operating System / Build Version

macOS Monterey 12.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Windows User Only

No response

Actual Behavior

When trying to launch a Postgres container with a bind mount, it doesn't work because of a chown related error to the folder at startup

Steps to Reproduce

Clone the repo https://github.com/docker/awesome-compose, go to the folder nginx-golang-postgres, edit the file docker-compose.yml to use a bind mount like the following

services:
  backend:
    build: backend
    secrets:
      - db-password
    depends_on:
      - db
  db:
    image: postgres
    restart: always
    secrets:
      - db-password
    volumes:
      - $PWD/db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=example
      - POSTGRES_PASSWORD_FILE=/run/secrets/db-password
    expose:
      - 5432

  proxy:
    build: proxy
    ports:
      - 8000:8000
    depends_on:
      - backend
#volumes:
#  db-data:
secrets:
  db-password:
    file: db/password.txt

Run the following command: docker compose up

Result

Error response from daemon: error while creating mount source path '~/github.com/docker/awesome-compose/nginx-golang-postgres/db-data': chown ~/github.com/docker/awesome-compose/nginx-golang-postgres/db-data: permission denied

Expected Behavior

I would expect to be able to access locally the folder as a bind mount in order to access and modify the files directly

Additional Information

No response

[guild-jonathan-kaczynski]

Rancher Desktop Version

0.7.1 and 1.0.0-beta.1

Rancher Desktop K8s Version

1.23.1 (latest), using the dockerd (moby) container runtime

What operating system are you using?

macOS

Operating System / Build Version

macOS Catalina 10.15.7 (19H1615)

What CPU architecture are you using?

x86-64 (Intel)

Windows User Only

No response

Actual Behavior

Trying to chown a folder mounted as a volume, from inside the container fails with a permission error.

Steps to Reproduce

Here is some of the output from running the endpoint script manually, if it helps any.

$ mkdir ./postgres

$ ls -ld ./postgres
drwxr-xr-x  2 jonathankaczynski  staff  64 Jan 24 13:38 ./postgres

$ docker run --rm -it \
    --entrypoint /bin/bash \
    -v "$(pwd)/postgres:/var/lib/postgresql/data" \
    postgres

root@98a1f91309fc:/# bash -x /usr/local/bin/docker-entrypoint.sh postgres
… snip …
+ docker_create_db_directories
+ local user
++ id -u
+ user=0
+ mkdir -p /var/lib/postgresql/data
+ chmod 700 /var/lib/postgresql/data
+ mkdir -p /var/run/postgresql
+ chmod 775 /var/run/postgresql
+ '[' -n '' ']'
+ '[' 0 = 0 ']'
+ find /var/lib/postgresql/data '!' -user postgres -exec chown postgres '{}' +

root@ff5dae1c266d:/# find /var/lib/postgresql/data '!' -user postgres
/var/lib/postgresql/data

root@ff5dae1c266d:/# ls -ld /var/lib/postgresql/data
drwx------ 1 501 dialout 64 Jan 24 18:32 /var/lib/postgresql/data

root@ff5dae1c266d:/# exit
exit

$ ls -ld ./postgres
drwx------  2 jonathankaczynski  staff  64 Jan 24 13:32 ./postgres

[guild-jonathan-kaczynski]

Here's a minimal test case derived from the above practical example.

There seem to be two potentially independent mounted volume errors.

The first error relates to the mounted volume not existing on the host os (macos) prior to running the docker command.

$ ls -ld ./foobar
ls: ./foobar: No such file or directory

$ docker run --rm -it -v "$(pwd)/foobar:/opt/foobar" debian:bullseye-slim
docker: Error response from daemon: error while creating mount source path '/Users/jonathankaczynski/foobar': chown /Users/jonathankaczynski/foobar: permission denied.

$ ls -ld ./foobar
drwxr-xr-x  2 jonathankaczynski  staff  64 Jan 24 14:50 ./foobar

$ docker run --rm -it -v "$(pwd)/foobar:/opt/foobar" debian:bullseye-slim

root@392147ccc4f5:/# exit
exit

The second error relates to attempting to change the ownership of the mount point within the docker container. Though changing the file mode succeeds.

$ mkdir ./foobar

$ docker run --rm -it -v "$(pwd)/foobar:/opt/foobar" debian:bullseye-slim

root@cc1c034865bd:/# ls -ld /opt/foobar
drwxr-xr-x 1 501 dialout 64 Jan 24 19:50 /opt/foobar

root@cc1c034865bd:/# groupadd -r postgres --gid=999

root@cc1c034865bd:/# useradd -r -g postgres --uid=999 postgres

root@cc1c034865bd:/# chown postgres /opt/foobar
chown: changing ownership of '/opt/foobar': Permission denied

root@cc1c034865bd:/# chmod 700 /opt/foobar

root@cc1c034865bd:/# exit
exit

$ ls -ld ./foobar
drwx------  2 jonathankaczynski  staff  64 Jan 24 14:50 ./foobar

[luiz290788]

I'm having the same problem, this is the only thing holding me from using Rancher Desktop. Any progress here?

[jeesmon]

MacOS, RD v1.0.0

Getting permission error when running postgres image with bind-mount (-v $(pwd):/var/lib/pgsql/data)

docker run --rm --name postgresql -e POSTGRESQL_DATABASE=my-db -e POSTGRESQL_USER=user -e POSTGRESQL_PASSWORD=pass -p 5432:5432 -v $(pwd):/var/lib/pgsql/data centos/postgresql-96-centos7

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... FATAL:  could not read block 2 in file "base/1/1255_fsm": read only 0 of 8192 bytes
PANIC:  cannot abort transaction 1, it was already committed
child process was terminated by signal 6: Aborted
initdb: removing contents of data directory "/var/lib/pgsql/data/userdata"

But if I use named volume (-v postgres-data:/var/lib/pgsql/data), it works fine

docker run --rm --name postgresql -e POSTGRESQL_DATABASE=my-db -e POSTGRESQL_USER=user -e POSTGRESQL_PASSWORD=pass -p 5432:5432 -v postgres-data:/var/lib/pgsql/data centos/postgresql-96-centos7

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
 done
server started
/var/run/postgresql:5432 - accepting connections
=> sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
ALTER ROLE
waiting for server to shut down.... done
server stopped
Starting server...
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".

docker volume list
DRIVER    VOLUME NAME
local     postgres-data

[petersondrew]

I think this may be an issue with the underlying lima. I experience the exact same behavior using either Rancher Desktop or colima.

Possibly related lima-vm/lima#504

[guild-jonathan-kaczynski]

They converted that issue into a discussion https://github.com/lima-vm/lima/discussions/505 marked it as answered.

sshfs isn't robust (and fast) enough to be used as /var/lib/postgresql. Please use lima nerdctl volume create to create a named volume inside the guest ext4 filesystem.

To me, it doesn't feel like the answer is addressing the broader concerns raised above.

[guild-jonathan-kaczynski]

There's also this earlier issue thread: https://github.com/lima-vm/lima/issues/231

The last comment, which is from Dec, was:

The plan is to use mapped-xattr or mapped-file of virtio 9P, but the patch is not merged for macOS hosts yet, and seems to need more testers: NixOS/nixpkgs#122420

[willcohen]

As a followup, the latest version of the patch is https://gitlab.com/wwcohen/qemu/-/tree/9p-darwin and that's where the in-progress work will go as it progresses towards resubmission upstream. Any comments on how to improve would be GREATLY welcomed before I submit again.

[guild-jonathan-kaczynski]

From https://github.com/NixOS/nixpkgs/pull/122420, it looks like good progress has been made:

9p-darwin has been merged upstream

I'm also going to close this PR in favor of https://github.com/NixOS/nixpkgs/pull/162243, at this point. I think it's okay if any additional final discussion still happens here since this particular issue has been referenced in so many places, but the work is now done!

[willcohen]

Please let me know if you have any questions!

[dennisdaotvlk]

Facing the same error with docker-compose and docker run -v

[marnen]

Still an issue with Rancher Desktop 1.4.1, exactly as described above. This is the only thing preventing me from using Rancher as opposed to Docker.

[laoyaoer]

Looks like I have encountered the same issue when trying to run gitlab. Unlike oxfs, all files belong to same uid 501 no matter on my Macbook or inside container. so the container keeps restarting and logging this error: directory[/etc/gitlab] (gitlab::default line 35) had an error: Errno::EACCES: Permission denied @ apply2files - /etc/gitlab

[ggustafsson]

This will (probably) be fixed when Lima bumps up to version 1.0.

$ ~/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl --version
limactl version 0.9.2-54-ge82db87

Why? Because 9P will be used instead of sshocker by default. Aka virtio-9p-pci in QEMU.

https://github.com/lima-vm/lima/issues/20

https://github.com/lima-vm/lima/blob/master/docs/mount.md

[ggustafsson]

FYI...

$ cat ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml
mountType: 9p

$ docker compose up
[+] Running 4/0
 ⠿ Network rabbitmq_default        Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq1-1  Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq3-1  Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq2-1  Created                                                                                                                                                                        0.0s
Attaching to rabbitmq-rabbitmq1-1, rabbitmq-rabbitmq2-1, rabbitmq-rabbitmq3-1
...
rabbitmq-rabbitmq3-1  |   Starting broker... completed with 4 plugins.
rabbitmq-rabbitmq1-1  |   Starting broker... completed with 4 plugins.
rabbitmq-rabbitmq2-1  |   Starting broker... completed with 4 plugins.

Switching over to 9P does indeed solve part of the problem for me. The chown issue disappears but it was replaced with file create issues in my case, changing to loose permissions on all dirs (777) resolved that however.

[laoyaoer]

FYI...

$ cat ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml
mountType: 9p

$ docker compose up
[+] Running 4/0
 ⠿ Network rabbitmq_default        Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq1-1  Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq3-1  Created                                                                                                                                                                        0.0s
 ⠿ Container rabbitmq-rabbitmq2-1  Created                                                                                                                                                                        0.0s
Attaching to rabbitmq-rabbitmq1-1, rabbitmq-rabbitmq2-1, rabbitmq-rabbitmq3-1
...
rabbitmq-rabbitmq3-1  |   Starting broker... completed with 4 plugins.
rabbitmq-rabbitmq1-1  |   Starting broker... completed with 4 plugins.
rabbitmq-rabbitmq2-1  |   Starting broker... completed with 4 plugins.

Switching over to 9P does indeed solve part of the problem for me. The chown issue disappears but it was replaced with file create issues in my case, changing to loose permissions on all dirs (777) resolved that however.

This will (probably) be fixed when Lima bumps up to version 1.0.

$ ~/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl --version
limactl version 0.9.2-54-ge82db87

Why? Because 9P will be used instead of sshocker by default. Aka virtio-9p-pci in QEMU.

lima-vm/lima#20

https://github.com/lima-vm/lima/blob/master/docs/mount.md

Thanks, I've already changed to Docker Desktop. Will install Rancher when this issue solved.

[leenooks]

I too am keen to see this resolved. Having been a Docker Desktop user, and developing on my M1 mac, I have no issues with permissions when starting docker containers. All my project files are owned by my id, however the docker containers that start (eg: Postgres or mysql) while they may attempt to chown and change permissions on files as part of their normal startup process - those attempts don't fail and those containers run happily.

On rancher desktop, those file mod permissions fail with permission denied errors - and if I change the uid/gid (from the host) to what the process is running on in the container, then those files are not visible and file not found errors pursue.

I changed from Docker Desktop on my company issued device, as I didn't want my company to think that I was using commercial software without meeting the license requirements, nor did I want Docker to think I was doing the same.

As much as I am glad that Rancher Desktop exists, ~~I cannot use it as a functional replacement to Docker Desktop :(~~

EDIT: I just tried the 9p in the override.yml as suggested above - and it does go along way to making Rancher Desktop a functional replacement to Docker Desktop. I had an issue with permissions, which I had to fix "in container" but once done, things were working better. :D

[guild-jonathan-kaczynski]

I see this discussion over on the lima project page https://github.com/lima-vm/lima/issues/971 (v1.0 roadmap: change the default mount driver from reverse-sshfs to 9p)

[ryancurrah]

Looks like 6 days ago Lima documented using virtiofs https://github.com/lima-vm/lima/commit/c18ae239b69a47db77436765b9b4861aaa0d595d

[jandubois]

Looks like 6 days ago Lima documented using virtiofs lima-vm/lima@c18ae23

It is still unreleased. Also will require macOS 13 (Ventura).

[yllekz]

Also having this problem but it's across more than postgres/mongo - random containers of mine are having chown-related issues like this. I attempted the 9p workaround but that made the problem even worse (all of my containers started chaotically crashing/restarting)

[jsoref]

You can include the following content in ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml:

mountType: 9p
mounts:
  - location: "~"
    9p:
      securityModel: mapped-xattr
      cache: "mmap"

It should allow this to work (you must restart Rancher Desktop to apply this setting).

Caveats: any symlinks on your host system will be seen as the referenced object in the VM/container. If there's a symlink loop, and something tries to follow it, it'll eat its own tail (potentially slowly depending on how things behave).

The databases I'm playing w/ (postgres, redis, neo4j) don't generally deal in symlinks, so I believe it's a satisfactory configuration for my database use cases. (It may create a mess for all of my other use cases, but that remains to be seen.)

[QuentinStouffs]

You can include the following content in ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml:

mountType: 9p
mounts:
  - location: "~"
    9p:
      securityModel: mapped-xattr
      cache: "mmap"

It should allow this to work (you must restart Rancher Desktop to apply this setting).

Caveats: any symlinks on your host system will be seen as the referenced object in the VM/container. If there's a symlink loop, and something tries to follow it, it'll eat its own tail (potentially slowly depending on how things behave).

The databases I'm playing w/ (postgres, redis, neo4j) don't generally deal in symlinks, so I believe it's a satisfactory configuration for my database use cases. (It may create a mess for all of my other use cases, but that remains to be seen.)

Thanks a lot, this solved my issue!

[Jay-Madden]

This ended up being a problem with colima on my m1 mac for me and ultimately with the lima vm, I had to upgrade to ventura and switch colima to vz vmtype and set my mountype to virtiofs then uninstall colima and lima and reinstall them and the mount worked

steps I took

1. run colima template and set vmType and mountType

# Virtual Machine type (qemu, vz)
# NOTE: this is macOS 13 only. For Linux and macOS <13.0, qemu is always used.
#
# vz is macOS virtualization framework and requires macOS 13
#
# Default: qemu
vmType: vz

# Utilise rosetta for amd64 emulation (requires m1 mac and vmType `vz`)
# Default: false
rosetta: false

# Volume mount driver for the virtual machine (virtiofs, 9p, sshfs).
#
# virtiofs is limited to macOS and vmType `vz`. It is the fastest of the options.
#
# 9p is the recommended and the most stable option for vmType `qemu`.
#
# sshfs is faster than 9p but the least reliable of the options (when there are lots
# of concurrent reads or writes).
#
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: virtiofs

2. uninstall and reinstall colima brew uninstall colima && brew uninstall lima
3. reinstall colima and start it up

brew install colima
colima start

[roy-t]

I encountered a similar issue where on Windows hosts the non-root user inside the container no longer had write access to mounts after switching from docker-desktop to rancher-desktop.

I'm using a docker-compose file with the following volume

volumes:
  - ${ROOT_DIRECTORY}/logs:/opt/something/logs

If the directory ${ROOT_DIRECTORY}/logs does not exist on the Windows Host it gets created when I run docker-compose up for the first time. The root user inside the container will be the owner of the /opt/something/logs directory and the non-root user inside the container will only have read access.

If the directory already existed on the Windows hosts (it is not created by Rancher-Desktop), the non root user inside the container also has write access.

I guess there's some difference in how Rancher-Desktop creates the folder on the host as compared to how Docker-Desktop did it on Windows hosts.

[jbilliau-rcd]

I'm trying to use Rancher Desktop with the simplest compose file ever and get a similar error:

version: '3'
services:
  redis:
    container_name: rmo-redis
    image: redis:6
    ports:
      - 6379:6379
    command: ['redis-server']
    volumes:
      - ./:/tmp

Running "docker compose up" gives me:

➜  ~/git/scratch/composetest/web git:(main) ✗ docker compose up
[+] Running 1/0
 ⠿ Container rmo-redis  Recreated                                                                                                                                                                      0.0s
Attaching to rmo-redis
rmo-redis  | chown: changing ownership of '.': Permission denied
rmo-redis exited with code 1

It seems like the only solution is to use the mountType: 9p solution described by @QuentinStouffs above, but according to this link - https://github.com/lima-vm/lima/issues/20#issue-895105285 (and our own devs), it's ridicoulously slow to the point of unusable.

[spawnrider]

Hi, Same issue for me on Jaeger/Prometheus with Rancher Desktop & Docker compose on Mac M1 :

[+] Running 2/0
 ⠿ Container jaeger-tracing-1     Created                                                                  0.0s
 ⠿ Container jaeger-prometheus-1  Created                                                                  0.0s
Attaching to jaeger-prometheus-1, jaeger-tracing-1
Error response from daemon: error while creating mount source path '/Users/xxx/Documents/Dev/Playground/opentelemetry/jaeger/prometheus.yml': chown /Users/xxx/Documents/Dev/Playground/opentelemetry/jaeger/prometheus.yml: permission denied```

[yangtze64]

How to solve this problem. My Rancher Desktop version is 1.9.0-tech-preview still have a similar problem.

[jsedano-emobg]

Any chance that this problem https://github.com/rancher-sandbox/rancher-desktop/issues/2514 is related to this one?

[jsoref]

@jsedano-emobg: sure?

Rancher Desktop (on macOS) uses limavm and the way that host files are mapped to the guest in limavm is not the same as the way that Docker Desktop (on macOS) does things. You can change how limavm does it (e.g. via 9p -- https://github.com/rancher-sandbox/rancher-desktop/issues/1209#issuecomment-1370181132), but it isn't a 100% feature for feature quirk for quirk implementation.