abiosoft
Trying to start a project in DDEV but getting an API error (500)
Description
I ran into something odd. I had a project running with ddev and colima 0.6.2 (on a m1pro with sonoma and vz/virtiofs). i wanted to start a second project in the terminal within vscode. i got the following output:
$> ddev start
Starting modi...
Failed to RunSimpleContainer to inspect database version/type: failed to create/start Docker container ({GetExistingDBType-modi-MxdvNY 0x14000220960 0x14000080480 <nil> <nil>}):API error (500): mkdir /var/lib/docker/overlay2/a784aef8fbf3af0f2a38904f816e31b88e268056b829a6b35028d77864d4a4c2-init: read-only file system, output=
when i checked the containers:
$> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
11c62cb4fdd6 ddev/ddev-traefik-router:v1.22.4 "/entrypoint.sh --co…" About an hour ago Up About an hour (healthy) 127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp, 127.0.0.1:8025-8026->8025-8026/tcp, 127.0.0.1:8036-8037->8036-8037/tcp, 127.0.0.1:10999->10999/tcp ddev-router
afa5b0a2bd88 phpmyadmin:5 "/docker-entrypoint.…" About an hour ago Up About an hour 80/tcp ddev-tests-phpmyadmin
6eeda14e67e0 ddev/ddev-dbserver-mariadb-10.5:v1.22.4-tests-built "/docker-entrypoint.…" About an hour ago Up About an hour (healthy) 127.0.0.1:32777->3306/tcp ddev-tests-db
120c00fe7e45 ddev/ddev-webserver:20231108_new_lagoon-tests-built "/pre-start.sh" About an hour ago Up About an hour (healthy) 8025/tcp, 127.0.0.1:32776->80/tcp, 127.0.0.1:32775->443/tcp ddev-tests-web
c71301f7966d ddev/ddev-ssh-agent:v1.22.4-built "/entry.sh ssh-agent" About an hour ago Up About an hour (healthy) ddev-ssh-agent```
the container listed on the failed startup for the project modi doesnt exist. i then tried to poweroff ddev
$> ddev poweroff
Failed removing custom certs/traefik configuration: failed to create/start Docker container ({poweroff-tHzHOH 0x1400021e1e0 0x1400020c480
i then tried a hard reset:
$> docker rmi -f $(docker images -q) Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3892998772: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1151577697: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3079715760: read-only file system Error response from daemon: conflict: unable to delete 1066e79ef629 (cannot be forced) - image is being used by running container 6eeda14e67e0 Error response from daemon: conflict: unable to delete a8483ea26b94 (cannot be forced) - image is being used by running container c71301f7966d Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1453494246: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3496979304: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3644803181: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3927001417: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json2578752554: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3753705229: read-only file system Error response from daemon: conflict: unable to delete 97c388ff3350 (cannot be forced) - image is being used by running container 11c62cb4fdd6 Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json389738583: read-only file system
and
$> docker rm -f $(docker ps -aq) Error response from daemon: Could not kill running container 11c62cb4fdd60e4793b2ce078dec0633f1134a1487f06e6009018caff80c1952, cannot remove - container 11c62cb4fdd6 PID 284811 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system Error response from daemon: Could not kill running container 6eeda14e67e032c9f59bba1c64a23cb80d17a4a082d65898113449619b8527cb, cannot remove - container 6eeda14e67e0 PID 280580 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes Error response from daemon: container 120c00fe7e45844a8c7860ce9d76ed41e891d4b7344665dbe8d466a99f472802: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/391d9432305b168f4a622f4927d74b55f046191a65201f2d27bbba67516fd78d: read-only file system Error response from daemon: container c71301f7966dfeaacda4248f1ce4dd2a98b0fe61d56fdae435ea531202bd0fca: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/d8fd6e612336f8b08c40b77cfa3afa9fbb49c3b6f1a5cd2cf756e64032d5750f: read-only file system```
i also tried to execute docker rm -f $(docker ps -aq)inside the colima vm after colima ssh same result like on the host.
no idea what caused it and or if you need any further informations?
Version
$> colima version && limactl --version && qemu-img --version
colima version 0.6.2
git commit: 22d7e5fbc86d5b8e3b27065a762800bc7960a0ff
runtime: docker
arch: aarch64
client: v24.0.7
server: v24.0.5
limactl version 0.18.0
qemu-img version 8.1.2
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
ddev version v1.22.4-37-gca6b43689
Operating System
- [ ] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [ ] Apple Silicon <= 13 (Ventura)
- [X] Apple Silicon >= 14 (Sonoma)
- [ ] Linux
Output of colima status
$> colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/rkoller/.colima/default/docker.sock
Reproduction Steps
I've outlined the steps that snowballed into the trouble. not sure if it is reproducible.
Expected behaviour
the error cascade happened out of the blue.
Additional context
No response
Can you try with the current development version?
brew install --HEAD colima
My bet is this is solved in 0.6.4
i think so too.
Can you try with the current development version?
testing might be difficult with the profile i ran into the issue. in the course of trying to recover or at least powerdown ddev i deleted some images and projects. and in the end the situation was worth. i've backup'ed and cloned the profile still based on the recommendation of @rfay but then deleted the original and created a new profile to being able to access some of the projects today. and i would have tested the HEAD version but now already a release is tagged and i gonna wait a few more hours until it shows up on homebrew and then give it a try. but as i said i also think the odds are high that things will be fixed with this release and i would also have a hard time to reproduce the issue anyway. i'Ve just reported it as a fyi that something like that happend and "maybe" you would have been able to notice any clues what has lead to that error. but will test 0.6.4 as soon as it is available on homebrew. thank you for the fast release!