devcontainers
Codespaces: "Cannot connect to the Docker daemon" occurs randomly (postCreateCommand)
Hi,
I'm trying to get DDEV - an open source Docker-based PHP development toolkit - working reliably in Codespaces. We already had a discussion here, there were some fixes for docker-in-docker in unversal image in the past (🙏 ) and @eiriksm (and others) provided a wait for docker script for the postCreateCommand, which I thought worked.
#!/bin/bash
set -ex
wait_for_docker() {
while true; do
docker ps > /dev/null 2>&1 && break
sleep 1
done
echo "Docker is ready."
}
wait_for_docker
# proceed with commands requiring docker
But for some Codespaces launches I still run into
"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?".
Other launches work fine as expected, current ratio is 5 working, 1 error - seems random.
My devcontainer.json:
{
"image": "mcr.microsoft.com/devcontainers/universal:2",
"features": {
"ghcr.io/ddev/ddev/install-ddev:latest": {}
},
// ...
"postCreateCommand": "chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh"
}
Example error output:
2024-05-18 08:23:04.024Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-05-18 08:23:04.171Z: + wait_for_docker
+ ********
+ docker ps
2024-05-18 08:23:04.666Z: + break
2024-05-18 08:23:04.670Z: + echo 'Docker is ready.'
Docker is ready.
+ ddev debug download-images
2024-05-18 08:23:04.829Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ... 2024-05-18 08:23:04.839Z:
2024-05-18 08:23:05.647Z:
docker-compose 0 B / 60.09 MiB [---------------------------------------] 0.00%2024-05-18 08:23:05.848Z:
docker-compose 25.86 MiB / 60.09 MiB [==============>------------------] 43.04%2024-05-18 08:23:06.048Z:
docker-compose 41.45 MiB / 60.09 MiB [======================>----------] 68.99%2024-05-18 08:23:06.235Z:
docker-compose 60.09 MiB / 60.09 MiB [==============================] 100.00% 0s
Download complete.
2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-05-18 08:23:20.078Z: Failed to PullBaseContainerImages(): exit status 1
2024-05-18 08:23:20.085Z: {"outcome":"error","message":"Command failed: /bin/sh -c chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh","description":"The postCreateCommand in the devcontainer.json failed.","containerId":"26560cae79c4541c929a677101cf0dea4a9632ceb233c8571c85425a124d2a54"}
2024-05-18 08:23:20.126Z: postCreateCommand failed with exit code 1. Skipping any further user-provided commands.
Full Log: https://gist.github.com/mandrasch/f0ddc7c275c8df68275c88bae5ec9424
My setup: https://github.com/mandrasch/ddev-codespaces-launch-blank Usage: Browser-based postCreateCommand: https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh
Is there a way to debug this? Thanks very much in advance for assistance! 🤗
I thought this had already been dealt with a time or two in previous issues?
Yes, there were fixes in docker-in-docker (universal image), but there is also a new spec for better lifecycle with docker in codespaces mentioned here: https://github.com/devcontainers/features/issues/634#issuecomment-1692306977
I'm not sure if this improvement would solve the random docker connection error I saw a few times now or if this is unrelated, therefore I opened up a new issue.
Hi 👋
Docker is ready. 2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
This doesn't look good, the wait_for_docker script thinks the docker started, but apparently that's not the case?
Can you update https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh as follows and see if this helps? Thanks!
# Docker can take a couple seconds to come up. Wait for it to be ready before
# proceeding with bootstrap.
iterations=10
while ! docker ps &>/dev/null; do
if [[ $iterations -eq 0 ]]; then
echo "Timeout waiting for the Docker daemon to start."
exit 1
fi
iterations=$((iterations - 1))
echo 'Docker is not ready. Waiting 10 seconds and trying again.'
sleep 10
done
Hi @samruddhikhandale, thanks very much for reply! Much appreciated!
I tested the new wait for docker script.
1 attempt out of 11 still failed: https://gist.github.com/mandrasch/1d293ba7ebe34aef6b8134a4637b29ad
In the failed attempt, docker ps seems to be responding immediately? There are no iterations, but it fails afterwards.
2024-06-06 15:52:38.406Z: + iterations=102024-06-06 15:52:38.419Z:
+ docker ps
2024-06-06 15:52:38.657Z: + ddev debug download-images2024-06-06 15:52:38.668Z:
2024-06-06 15:52:53.999Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-06-06 15:52:54.004Z: Failed to PullBaseContainerImages(): exit status 1
The sucessful attempts seem to run through some iterations before docker ps responds:
2024-06-06 15:53:07.738Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-06-06 15:53:07.850Z: + iterations=10
+ docker ps
2024-06-06 15:53:19.330Z: + [[ 10 -eq 0 ]]2024-06-06 15:53:19.333Z:
+ iterations=9
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:29.346Z: + docker ps
2024-06-06 15:53:39.589Z: + [[ 9 -eq 0 ]]
2024-06-06 15:53:39.598Z: + iterations=8
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:49.604Z: + docker ps
2024-06-06 15:53:49.641Z: + ddev debug download-images
2024-06-06 15:53:49.726Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ...
Log of successful attempt: https://gist.github.com/mandrasch/8b296552af9c66a1acc4036a10f1154b
Thanks for getting back to us. @gauravsaini04 Can you help investigate the issue? It would be good to find out what could be causing the docker to stop after getting started.
Hi @samruddhikhandale, I was not able to replicate the issue. Tried approx. 20 - 25 times and it would pick up docker almost instantly and doesn't seem to be failing afterwards as well. @mandrasch, let me know if you are still facing the issue with docker daemon stopping randomly.
Potentially unrelated, but I am attempting to start a supabase docker container in a github codespace prebuild.
- I'm not sure that is possible or proper usage of a prebuild
- If it is possible I'm not sure which build step it should occur in.
I currently have this as my dev container config. My main goal is to avoid having to wait on docker registry downloads when running supabase start after first creation of a codespace. However, since this should just be a one time build time for the first creation of the the codespace it probably isn't work including in the prebuild
{
"image": "mcr.microsoft.com/devcontainers/universal:2",
"features": {
"ghcr.io/devcontainers-extra/features/supabase-cli:1": {}, // Supabase CLI installed
"ghcr.io/devcontainers-community/npm-features/prettier:1": {} // Prettier feature
},
"onCreateCommand": "npm install && supabase start", // Install dependencies and start frontend after Codespace is created
"postStartCommand": "supabase start && npm run dev" // Ensure Supabase and frontend run on every start
}
same here.
my devcontainer.json
{
"name": "Mili Container",
"image": "mcr.microsoft.com/devcontainers/universal:2",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {}
},
"customizations": {
"vscode": {
"extensions": [],
"settings": {
"editor.formatOnSave": true
}
}
},
"postStartCommand": "cd /workspaces/mili/ && docker compose up -d"
}
log:
/usr/local/share/pull-git-lfs-artifacts.sh
Fetching git lfs artifacts...
Running the postStartCommand from devcontainer.json...
cd /workspaces/mili/ && docker compose up -d
unable to get image 'grafana/grafana': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[931 ms] postStartCommand failed with exit code 1. Skipping any further user-provided commands.
{"outcome":"error","message":"Command failed: /bin/sh -c cd /workspaces/mili/ && docker compose up -d","description":"The postStartCommand in the devcontainer.json failed.","containerId":"abed77b7f8770a1ecca83a1d58b887eb49c5988984f8eee4d969bad44c9b4c11"}
