Unable to write to bind mount

[spuder]

colima version 0.2.2
git commit: b2c7697bee2d73e995f156fe8e9870eb246c07e6

runtime: docker
client: v20.10.12
server: v20.10.7

uname -a
Darwin <REDACTED> 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

When bind mounting a volume on the host, the filesystem is read only despite using docker functionality to allow for writing to the host

Steps to reproduce

Note that :Z is added to the -v parameter.

docker run -v "$(PWD):/output:Z"  bash:latest touch /output/foo.txt
touch: cannot touch '/output/foo.txt': Read-only file system

I've also tried 4. docker run -v "$(PWD)/output:z" bash touch /output/foo.txt 4. docker run -v "$(PWD)/output:rw" bash touch /output/foo.txt 3. docker run --privileged -v "$(PWD):/output:Z" bash touch /output/foo.txt 4. docker run -v "$(PWD)/output:rw" --group-add=dialout bash touch /output/foo.txt 5. sudo docker run -v "$(PWD)/output:rw" --group-add=dialout bash touch /output/foo.txt

With the same error

Files appear to be owned by 502:dialout

docker run -v "$(PWD)/CAD:/output:rw" bash ls -al /output
total 260
-rw-r--r-- 1 502 dialout  50758 Jan  7 02:48 MPB2015 Feet v4.f3d
-rw-r--r-- 1 502 dialout 212684 Jan  7 02:42 MPB2015-Feet.stl

Is this a known limitation? A bug? Or a misconfiguration on my part? Any guidance appreciated.

[spuder]

I suspect this is a lima problem

https://github.com/lima-vm/lima/issues/34#issuecomment-928808005

Looking at ~/.lima/default/lima.yaml` the default settings are:

# Expose host directories to the guest, the mount point might be accessible from all UIDs in the guest
# Default: none
mounts:
  - location: "~"
    # CAUTION: `writable` SHOULD be false for the home directory.
    # Setting `writable` to true is possible, but untested and dangerous.
    writable: false
  - location: "/tmp/lima"
    writable: true

I don't know if this quite translates to the problem that I am seeing, but it does seem like a plausible correlation.

[abiosoft]

@spuder version 0.2.2 do not mount home directory as writeable by default, kindly install the latest development version with brew install --head colima.

[spuder]

colima stop
colima delete
brew unlink colima
brew install --head colima
colima start

While I no longer get the error Read-only file system, I'm finding that I'm unable to write to the host

docker run -v "$(PWD):/output" bash ls /output
docker run -v "$(PWD):/output" bash touch /output/foo.txt
ls | grep foo.txt 
# no results

[spuder]

I've uncovered some sort of race condition, because I'm now able to touch a file once, but if I delete the file and try again I always get an error

❌

rm foo.txt
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | grep foo.txt
foo.txt <------ notice that file exists 
rm foo.txt
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
touch: /output/foo.txt: No such file or directory

Does lima\colima sync the file system periodically? I find that If I put a sleep between the touch statements, eventually I get to the point where it can write to the volume consistently

If I write a file with a unique name, it always works

✅

docker run --rm -v "$(PWD):/output" bash touch "/output/$(date).txt"
docker run --rm -v "$(PWD):/output" bash touch "/output/$(date).txt"
ls 
Mon Jan 10 16:22:33 MST 2022.txt   Mon Jan 10 16:22:40 MST 2022.txt

It seems like when you create/delete a file with lima/colima it is caching that file for a few seconds so you can't immediately write a file with the same name.

[jphenow]

This is sounding similar #65 maybe? I'm experiencing things more like #65 but the two sound very similar.

[jphenow]

I've been tinkering with stuff around this issue to see if I can get any better results but have been as yet unsuccessful

[yinshangwei]

When mounting Postgres data, they will be an error. Seems like the same issue. chown: changing ownership of '/var/lib/postgresql/data': Permission denied

[vraravam]

Yes - I too have the same issue with the following docker-compose.yml file:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - ./postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

I'm using the latest HEAD version of colima along with the above.

[abiosoft]

@spuder kindly update colima and lima and try again.

brew upgrade lima colima

[spuder]

I just upgraded colima and lima, but I"m not seeing that bind mounts don't work at all

brew upgrade lima colima
colima version 
  colima version 0.3.1
  git commit: 787ae5631ae8de072feef95a509c47fc93308b2e
  
  runtime: docker
  arch: x86_64
  client: v20.10.12
  server: v20.10.11
colima stop
colima delete
colima start
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | wc -l
0 <-------------- This should say 1 because there should be 1 file named foo.txt in the directory

I've reproduced this twice now and I'm pretty sure this is a regression

[john-ot]

I'm able to reproduce the issue as well. I'm running: colima start --cpu 2 --memory 8 --disk 40 --mount $HOME/IdeaProjects:/IdeaProjects:rw

But when I look at ~/.lima/colima/lima.yaml I see this:

mounts:
    - location: /Users/john/Library/Caches/colima
      writable: false
    - location: /Users/john/IdeaProjects/
      writable: false

I'm running colima 0.3.2

[abiosoft]

I'm able to reproduce the issue as well. I'm running: colima start --cpu 2 --memory 8 --disk 40 --mount $HOME/IdeaProjects:/IdeaProjects:rw

@john-ot You do not need to specify the mount actually, your $HOME directory is mounted by default, get rid of the mount flag and try again, you should not have any problems. Nonetheless, your mount syntax is wrong, it should be --mount $HOME/IdeaProjects:w.

[john-ot]

Ack! I will fix the syntax.

Does the default mount give me write access though? I thought it was read only by default?

[abiosoft]

Does the default mount give me write access though? I thought it was read only by default?

Used to be read-only, but changed to writable in version 0.3.0.

[hungchai]

YES ! I have same issue too. Can anyone look into it? OS:

colima status                                                          
INFO[0000] colima is running
INFO[0000] runtime: docker
INFO[0000] arch: x86_64

colima version                                                
colima version HEAD-784cec0
git commit: 784cec04dc126b02f5d2697a864282d5d2301afe

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11

docker-compose up redis                                             
[+] Running 2/2
 ⠿ Network foundation_default    Created                                                                                                             0.1s
 ⠿ Container foundation-redis-1  Created                                                                                                             0.1s
Attaching to foundation-redis-1
foundation-redis-1  | chown: changing ownership of '.': Permission denied
foundation-redis-1  | chown: changing ownership of './.DS_Store': Permission denied
foundation-redis-1  | chown: changing ownership of './dump.rdb': Permission denied
foundation-redis-1  | chown: changing ownership of './temp-1.rdb': Permission denied
foundation-redis-1 exited with code 1

docker-compose

  redis:
    image: redis
    ports:
      - "6379:6379"
    volumes:
      - ./redis/data:/data

lima.yaml

...
disk: 60GiB
mounts:
    - location: /Users/xxxx/Library/Caches/colima
      writable: false
    - location: /Users/xxxx/docker/
      writable: true

[DmitriiMukhin]

Yes - I too have the same issue with the following docker-compose.yml file:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - ./postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

I'm using the latest HEAD version of colima along with the above.

I'm facing the same issue.

[phaus]

Just a quick update:

I was able to solve this for some use cases. You need to use volumes instead of shared folders:

e.g. for the last docker-compose file use the volume postgres-data as follows:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

So use postgres-data:/var/lib/postgresql/data:Z instead of ./postgres-data:/var/lib/postgresql/data:Z

This issue is REALLY intereseting, becaus the error only happens, if a container tries to perform a chown to a folder shared via ssh-fs (what most DB Containers tend to do). In my case, I changed to docker volumes for my DB container and was able to write to a shared folder via my dev container while building a specific docker image with the access rights of my current user:

build script (USER and UID are already set via the shell):

#!/bin/bash

VERSION=$(git describe --always --dirty --tags)
CONTAINER=$(basename $PWD)
GID=$(id -g)

echo "building for $USER ($UID:$GID)"
export UID
export GID
export USER

docker build --build-arg USER --build-arg UID --build-arg GID -t ${CONTAINER}:${VERSION} .

Dockerfile:

FROM ghcr.io/baosystems/postgis:13-3.1

ARG USER
ARG UID
ARG GID

RUN adduser -u ${UID} --gid ${GID} ${USER}

ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    curl \
    jq \
    && rm -rf /var/lib/apt/lists/*

[DmitriiMukhin]

Just a quick update:

I was able to solve this for some use cases. You need to use volumes instead of shared folders:

e.g. for the last docker-compose file use the volume postgres-data as follows:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

So use postgres-data:/var/lib/postgresql/data:Z instead of ./postgres-data:/var/lib/postgresql/data:Z

This issue is REALLY intereseting, becaus the error only happens, if a container tries to perform a chown to a folder shared via ssh-fs (what most DB Containers tend to do). In my case, I changed to docker volumes for my DB container and was able to write to a shared folder via my dev container while building a specific docker image with the access rights of my current user:

build script (USER and UID are already set via the shell):

#!/bin/bash

VERSION=$(git describe --always --dirty --tags)
CONTAINER=$(basename $PWD)
GID=$(id -g)

echo "building for $USER ($UID:$GID)"
export UID
export GID
export USER

docker build --build-arg USER --build-arg UID --build-arg GID -t ${CONTAINER}:${VERSION} .

Dockerfile:

FROM ghcr.io/baosystems/postgis:13-3.1

ARG USER
ARG UID
ARG GID

RUN adduser -u ${UID} --gid ${GID} ${USER}

ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    curl \
    jq \
    && rm -rf /var/lib/apt/lists/*

Please provide a solution - how to create volumes that will be mapped to the host directory. I mean to $HOME/some/directory/postgres-data.

[jklewa]

When mounting Postgres data, they will be an error. Seems like the same issue. chown: changing ownership of '/var/lib/postgresql/data': Permission denied

chown: changing ownership of 'foo': Permission denied is a separate issue (https://github.com/abiosoft/colima/issues/83) than the read-only file system issue discussed in the beginning of this thread. Unfortunately, chown on a host-mounted directory isn't yet supported (upstream issue https://github.com/lima-vm/lima/issues/231, further upstreamed by https://github.com/NixOS/nixpkgs/pull/122420)

@yinshangwei cc @vraravam @hungchai @DmitriiMukhin

@phaus suggested the workarounds 1) to use a VM-only volume instead of a host-mounted directory, or 2) don't use chown by making the user:group IDs of those paths match between the VM and host

[phaus]

so from my side, this issue can be closed.

[HansG89]

any ideas on how to fix this issue when running local k3d kubernetes in colima and providing a host path storage to a pod?

Colima is running with default settings, so home direcotry is mounted as writable. ALso told k3d to mount the home directory. I tested touching a file to a mounted volume inside a pod and it works as expected. Only chowning a folder seems to cause trouble.

So for instance the postgres example gives me: chown: changing ownership of '/var/lib/postgresql/data/pgdata': Permission denied

config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: basys
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: admin
  POSTGRES_PASSWORD: test123
  PGDATA: /var/lib/postgresql/data/pgdata

storage:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/Users/xxx/postgres/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:14.1
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
              readOnly: false
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim

[phaus]

@HansG89 maybe you can use a light storage provider (like https://rook.io or https://longhorn.io) to create k8s persistent volumes? Otherwise is should also be possible to use NFS or other network storage option for k8s volumes. Using local mount paths in k8s is always a dev workaround IMHO, because in production you almost always go with persistent volumes.

[HansG89]

@HansG89 maybe you can use a light storage provider (like https://rook.io or https://longhorn.io) to create k8s persistent volumes? Otherwise is should also be possible to use NFS or other network storage option for k8s volumes.

Using local mount paths in k8s is always a dev workaround IMHO, because in production you almost always go with persistent volumes.

Thank you for your ideas. I used k3d to keep the setup as simple as possible. I don't want to introduce another level of complexity because a developer, who should use the setup in his daily business should be able to focus on developing features and not on infrastructure. This is why the host path solution is preferred. Also it's almost the same like the good old docker-compose setup. Anyways would be great if root cause in NixOS gets fixed as soon as possible 😉

[Cerebus]

Try mount --make-shared on the relevant VM mount point.

[iambudi]

I just upgraded colima and lima, but I"m not seeing that bind mounts don't work at all

brew upgrade lima colima
colima version 
  colima version 0.3.1
  git commit: 787ae5631ae8de072feef95a509c47fc93308b2e
  
  runtime: docker
  arch: x86_64
  client: v20.10.12
  server: v20.10.11
colima stop
colima delete
colima start
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | wc -l
0 <-------------- This should say 1 because there should be 1 file named foo.txt in the directory

I've reproduced this twice now and I'm pretty sure this is a regression

I'm having this difficulty as well. colima version 0.5.2. I need to mount rw to try hot reloading.

[iambudi]

It's working now after running colima delete and colima start

[berniedurfee-renaissance]

Just to clarify, this issue is still open and Colima doesn't allow for changing ownership of files in host mounted directories?

I'm seeing host mounted directories and files being owned by 506:dialout and need to change the ownership of those files and directoris.

I'm using the short form for mounting volumes:

...
services:
  my_service:
	...
	volumes:
      - my_files:/storage/my_files
...

Is this still expected behavior until this issue is resolved?

chown: changing ownership of '/storage/my_files': Permission denied

[rfay]

By default only your home directory is set up as writable. You'll need to adjust your colima.yml to make the mount that you want writable.