Issue with writable volumes

[matthieugouel]

trafficstars

Hi 👋

I just came across this project and it seems very promising, thanks for your work!

I have a little issue while setup a docker container with writable volume (it's a database).

I start Colima and I mount the local directory:

colima start --mount /Users/matthieugouel/Documents/Code:rw

Then, I try to create the docker container:

 docker run  --name clickhouse -v $(pwd)/volumes/clickhouse:/var/lib/clickhouse -p 9000:9000 yandex/clickhouse-server:latest

But the issue is:

chown: changing ownership of '/var/lib/clickhouse/': Read-only file system

It's strange because when I ssh inside the VM I have these rights for the volume directory:

drwxr-xr-x 1 matthieugouel dialout  352 Nov  7 13:45 volumes

Would you have an idea of why this issue is happening?

Best, Matthieu.

[abiosoft]

It is :w not :rw.

[matthieugouel]

In this case I have:

chown: changing ownership of '/var/lib/clickhouse/': Permission denied

[dustinrue]

This might be because the directory exists inside the colima VM. Probably the safest thing is to delete the colima VM you created and create it again with the corrected :w option

[matthieugouel]

I cleaned up everything with:

colima stop
colima delete
rm -Rf volumes/clickhouse

Then I started again the VM with:

colima start --mount /Users/matthieugouel/Documents/Code:w

I ssh onto the VM to check the state of the directories. At this point the directory volumes is there with the right permissions and the directory clickhouse is not there as intended.

I then created the clickhouse container:

docker run  --name clickhouse -v $(pwd)/volumes/clickhouse:/var/lib/clickhouse:rw -p 9000:9000 yandex/clickhouse-server:latest

I get the error:

docker: Error response from daemon: error while creating mount source path '/Users/matthieugouel/Documents/Code/research/iris/volumes/clickhouse': chown /Users/matthieugouel/Documents/Code/research/iris/volumes/clickhouse: permission denied.

If I check on the VM the directory has been created:

drwxr-xr-x 1 matthieugouel dialout   64 Nov  8 11:19 clickhouse

If I start again recreating the container without touching anything I get simply this error:

chown: changing ownership of '/var/lib/clickhouse/': Permission denied

[matthieugouel]

When I start the clickhouse container without any volume and I open a terminal on it, I remark that the directory I try to mount as a volume /var/lib/clickhouse has this permissions and user:

drwxrwxrwx 5 clickhouse clickhouse 4.0K Nov  9 22:40 clickhouse

Maybe it has to do with the fact that there is no clickhouse user on the VM and thus it gets a permission denied?

[abiosoft]

@matthieugouel do you get the same behaviour if the clickhouse directory exists before starting the container?

[matthieugouel]

Yes unfortunately. The issue seems to be on all images that use a specific user (for instance here clickhouse with the UID 101) that does not exist on the VM.

[abiosoft]

@matthieugouel can you try with the latest development version ? brew install --HEAD colima.

Then colima delete and start afresh. Kindly update if the behaviour is same.

[matthieugouel]

Hi! Exactly the same issue as before unfortunately.

[jjeising]

Adding

user: "<uid>:<gid>"

for docker-compose (or --user "<uid>:<gid>" for docker) seems to fix this for me. You can look these up via id.

[myou11]

Adding

user: "<uid>:<gid>"

for docker-compose (or --user "<uid>:<gid>" for docker) seems to fix this for me. You can look these up via id.

thanks for this @jjeising! To give more context as to why this works, in the Runtime section of the readme, it states "On initial startup, Colima initiates with a user specified runtime that defaults to Docker." I believe this causes the permission issues when using a bind mount since the bind mount files are owned by the host user while the colima VM is using the Docker user to spin up the container. By specifying the user key like @jjeising mentions, this will spin up the container with your host user which matches the owner of the bind mount and no permission issues occur. If there's any issues with this explanation, let me know and I can correct it. Just wanted to provide more clarity into why this works for future readers 😄

[simonedavico]

Adding

user: "<uid>:<gid>"

for docker-compose (or --user "<uid>:<gid>" for docker) seems to fix this for me. You can look these up via id.

Tried to set this in my compose file for a postgres service, but I still can't mount a volume :(

[luiz290788]

I tried the user configuration and couldn't do it as well. I got the user id and group id from id command and got this:

initdb: could not look up effective user ID 502: user does not exist

[andersjanmyr]

Example with mysql

docker run -v $PWD/.mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=example mysql

This generates a bunch of permissions errors.

2022-02-07 13:52:58+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.28-1debian10 started.
chown: changing ownership of '/var/lib/mysql/': Permission denied
...

Passing in the UID and GID results in a different error

docker run -v $PWD/.mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=example --user 1531978419:619956085 mysql
...
2022-02-07T13:56:23.919163Z 2 [ERROR] [MY-013183] [InnoDB] Assertion failure: rec.cc:271 thread 139630266889984
13:56:23 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x562bd4e58280
...

[gustavodecarlo]

I do a test with postgres in this way:

colima start --mount ~/bigdata/postgres_data:w

PWD=~/bigdata/postgres_data

docker run -v $PWD:/var/lib/postgresql/:rw -e POSTGRES_PASSWORD=password postgres

Result:

~/bigdata/postgres_data  $ docker run -v $PWD:/var/lib/postgresql/:rw -e POSTGRES_PASSWORD=password postgres postgres 
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2022-02-09 18:02:50.021 UTC [48] LOG:  starting PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-02-09 18:02:50.024 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-09 18:02:50.029 UTC [49] LOG:  database system was shut down at 2022-02-09 18:02:49 UTC
2022-02-09 18:02:50.033 UTC [48] LOG:  database system is ready to accept connections
 done
server started

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2022-02-09 18:02:50.154 UTC [48] LOG:  received fast shutdown request
waiting for server to shut down....2022-02-09 18:02:50.157 UTC [48] LOG:  aborting any active transactions
2022-02-09 18:02:50.158 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
2022-02-09 18:02:50.159 UTC [50] LOG:  shutting down
2022-02-09 18:02:50.173 UTC [48] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-02-09 18:02:50.282 UTC [1] LOG:  starting PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-02-09 18:02:50.282 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-09 18:02:50.282 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-02-09 18:02:50.287 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-09 18:02:50.293 UTC [60] LOG:  database system was shut down at 2022-02-09 18:02:50 UTC
2022-02-09 18:02:50.298 UTC [1] LOG:  database system is ready to accept connections

The host folder:

 ~/bigdata/postgres_data  $ ls -la
total 0
drwxr-xr-x  3 gustavodecarlotw  staff   96 Feb  9 15:02 .
drwxr-xr-x  5 gustavodecarlotw  staff  160 Feb  9 14:57 ..
drwxr-xr-x  2 gustavodecarlotw  staff   64 Feb  9 15:02 data

[kaminek]

same issue here, I get a permission denied on shared files.

[ghost]

Based on the info from gustavodecarlo this is what I ran:

colima delete rm -rf ~/Developer/postgres_data mkdir -p ~/Developer/postgres_data colima start --mount ~/Developer/postgres_data:w PWD=~/Developer/postgres_data docker run -d --name postgres -v $PWD:/var/lib/postgresql/:rw -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -e POSTGRES_PASSWORD=postgres postgres:13.6

but

ls -la ~/Developer/postgres_data/data total 0 drwxr-xr-x 2 codrut staff 64 Mar 31 01:46 . drwxr-xr-x 3 codrut staff 96 Mar 31 01:46 ..

So, there is no actual file sharing

[AndrewRayCode]

I have a docker for mac / docker-compose setup including a postgers container that persists data to a volume. I just installed colima and tried docker-compose up in my project folder. I got lots of:

Creating andrewrayme_db_1    ... done
Attaching to andrewrayme_db_1, andrewrayme_ghost_1
db_1     | chown: changing ownership of '/var/lib/postgresql/data': Permission denied
db_1     | chown: changing ownership of '/var/lib/postgresql/data/pg_multixact': Permission denied
db_1     | chown: changing ownership of '/var/lib/postgresql/data/pg_multixact/members': Permission denied
db_1     | chown: changing ownership of '/var/lib/postgresql/data/pg_multixact/members/0000': Permission denied
db_1     | chown: changing ownership of '/var/lib/postgresql/data/pg_multixact/offsets': Permission denied
db_1     | chown: changing ownership of '/var/lib/postgresql/data/pg_multixact/offsets/0000': Permission denied

Is this possibly a colima issue, or an issue switching existing volumes from docker for mac to colima?

[abiosoft]

Yeah, sadly it's a known issue with sshfs. You cannot change the owner of the mounted directory.

You can however change the owner of the subdirectories. So, specifying custom PGDATA may work as seen in the official Postgres image docs.

If you do not rely on symlinks and not primarily working off the mounted volumes, you can try the 9p mount.

colima start --mount-type 9p

[freeformz]

FWIW: I have this same problem with 9p volumes

[varzy]

Same problem with 9p.

[cbbntla]

same issue, I solved it with these steps:

1. stop colima and delete vm
2. re-run colima with colima start --edit, set the dictionary writable:true, which contains my docker-compose file and config file

mounts:
  - location: <my path>
    writable: true

[abiosoft]

If you are on macOS 13 or newer, you can give the currently development version a try.

brew install --HEAD colima 

[matthieugouel]

Hi @abiosoft,

I tested the new version of Colima (installed with the brew command line you suggested) on MacOS Ventura. I did the same test than my original message and it worked like a charm!

To me my issue is resolved, but others should test their own use case as well.

[varzy]

Same permission issues on MacOS Ventura 13.0.1, I try:

brew uninstall colima
brew install --HEAD colima
colima start --memory 4 --mount-type 9p
docker compose up -d mysql redis

Works for me.

[Lan-Phan]

Yeah, sadly it's a known issue with sshfs. You cannot change the owner of the mounted directory.

I got same issue with virtiofs mount type, and it's able to change owner of sub-directories.

[archcst]

Hi @abiosoft,

I tested the new version of Colima (installed with the brew command line you suggested) on MacOS Ventura. I did the same test than my original message and it worked like a charm!

To me my issue is resolved, but others should test their own use case as well.

Hi, I had the same issue, wondering were you using 9p or default sshfs after --HEAD installed?

[archcst]

I'v tried these command below:

❯  colima delete
❯  brew uninstall colima
❯  brew install --HEAD colima
❯  colima start --edit  # modified mountType to 9p, mount a writable path, modify runtime to containerd
INFO[0000] editing in vim
INFO[0387] starting colima
INFO[0387] runtime: containerd+k3s
INFO[0387] preparing network ...                         context=vm
INFO[0390] creating and starting ...                     context=vm
INFO[0421] provisioning ...                              context=containerd
INFO[0421] starting ...                                  context=containerd
INFO[0426] provisioning ...                              context=kubernetes
INFO[0426] downloading and installing ...                context=kubernetes
INFO[0434] loading oci images ...                        context=kubernetes
INFO[0438] starting ...                                  context=kubernetes
INFO[0441] updating config ...                           context=kubernetes
INFO[0441] Switched to context "colima".                 context=kubernetes
INFO[0442] done

❯  k get po -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
NAMESPACE     NAME                                      READY   STATUS             RESTARTS       AGE
kube-system   local-path-provisioner-79f67d76f8-g9vzh   1/1     Running            0              8m50s
kube-system   coredns-597584b69b-s88rf                  1/1     Running            0              8m50s
kube-system   helm-install-traefik-crd-m5wh5            0/1     Completed          0              8m50s
kube-system   svclb-traefik-8b4b7b98-sgjt5              2/2     Running            0              8m46s
kube-system   helm-install-traefik-t6nkr                0/1     Completed          1              8m50s
kube-system   traefik-bb69b68cd-vhrxj                   1/1     Running            0              8m46s
kube-system   metrics-server-5c8978b444-nhtwc           1/1     Running            0              8m50s

❯  kubectl apply -f mysql.yaml

❯  k logs -n middleware  mysql-6b86964546-q92c7
2023-02-28 12:57:26+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
chown: changing ownership of '/var/lib/mysql': Operation not permitted

[archcst]

#https://github.com/abiosoft/colima/issues/83#issuecomment-1339269542

fixed this one