Using VS Code Devcontainers (docker-from-docker) with Non Root User Failing Docker Daemon Credentials Check

[ghost]

• VSCode Version: Version: 1.74.2 Commit: e8a3071ea4344d9d48ef8a4df2c097372b0c5161 Date: 2022-12-20T10:32:21.587Z Electron: 19.1.8 Chromium: 102.0.5005.167 Node.js: 16.14.2 V8: 10.2.154.15-electron.0 OS: Darwin arm64 21.6.0 Sandboxed: No

• Local OS Version: Mac OS X Version 12.6

• Local chip architecture: <x86, arm64, Apple Silicon> Apple M1

• Reproduces in: <Codespaces | Remote - Containers | Both> Remote Containers

• Name of Dev Container Definition with Issue:

old location: https://github.com/microsoft/vscode-dev-containers/tree/main/containers/docker-from-docker

new location: https://github.com/devcontainers/features/tree/main/src/docker-from-docker

The entry from our devcontainer.json:

    "docker-from-docker": {
      "version": "latest",
      "moby": true
    },

I also tried updating the above to the "new format":

"features": {
    "ghcr.io/devcontainers/features/docker-from-docker:1": {}
}

to no effect.

I posted a question here: https://github.com/devcontainers/community/discussions/20

to find out where this issue/bug should be created - but this seems like the best place for it.

Steps to Reproduce:

Hi there, we've been successfully using devcontainers (non root user spinning up docker containers) for over a year now.

A new version of docker was pushed out in late December - with a new version of credential helper and it appears to have broken our usage of visual studio code devcontainer's. Our usage of devcontainers is with a non root user using the "docker from docker" workflow.

This technique has been working for over a year now using the documentation here: https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user

We added our non root user to the group docker.

When the issue first started happening we got the following error:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: rpc error: code = Unknown desc = error getting credentials - err: exit status 255, out: `

So we blew away all data (containers, images + volumes) using the docker purge data.

Then when we re-ran docker-compose - we started getting python errors like this:

Pulling <docker image name> (<docker image name>:)...
Traceback (most recent call last):
  File "/usr/local/pipx/venvs/docker-compose/lib/python3.9/site-packages/docker/credentials/store.py", line 76, in _execute
    output = subprocess.check_output(
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/docker-credential-dev-containers-f90694ffd4ef7d23592ec24c6172cf511d70ae8b', 'get']' returned non-zero exit status 255.

Along with a few other python errors - which appear to be similarly just trying to run some command line executables.

After a lot of reading online - it seems the issue is that the

/home/<non root user>/.docker/config.json

is getting populated with invalid credentials in the following format:

{
        "credsStore": "dev-containers-<hash number>"
}

After removing this entry from the config.json - everything "just works" - the non root user is once again able to talk to the docker daemon, create images + subsequently containers.

I'm not sure if this issue needs to be "cross posted" to a docker issue repository since this may be in the realm of an integration issue.

[samruddhikhandale]

Hi 👋

Thanks for the information.

Few clarifying questions,

1. What is the version of your dev containers Extension? Can you update to latest and retry?
2. Is your devcontainer same as the one defined in https://github.com/microsoft/vscode-dev-containers/tree/main/containers/docker-from-docker ? If yes, then do you mind switching to a simpler one below 👇

{
	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
	"image": "mcr.microsoft.com/devcontainers/base:bullseye",

	"features": {
                // docker-from-docker was renamed yesterday 👇 
		"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
			"version": "latest",
			"enableNonRootDocker": "true",
			"moby": "true"
		}
	},

	// Use this environment variable if you need to bind mount your local source code into a new container.
	"remoteEnv": {
		"LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}"
	}
}

3. If nothing of the above works, can you provide me with a repro (like you dev config)?

[ghost]

1. devcontains version: v0.269.0
2. I actually mentioned that in my original ticket - that we tried the new format (to no effect)
3. Unfortunately - I don't think I can publicly share the entirety of our devcontainer.json + Dockerfile - but I can share snippets of it. So as mentioned we've tried both (in our devcontainer.json):

  "features": {
    "docker-from-docker": {
      "version": "latest",
      "moby": true
    },

and the updated format:

"features": {
    "ghcr.io/devcontainers/features/docker-from-docker:1": {}
}

I believe the values you populated in your config snippet are actually the defaults - i.e. works the same as empty object afaik.

I can also mention this is what we're using in our Dockerfile:

ARG VARIANT="buster"
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}

Looking at the devcontainer.json here: https://github.com/microsoft/vscode-dev-containers/blob/v0.245.2/containers/debian/.devcontainer/devcontainer.json

and the Dockerfile here: https://github.com/microsoft/vscode-dev-containers/blob/v0.245.2/containers/debian/.devcontainer/Dockerfile

I can say that our Dockerfile (as seen above) differs from the version in the microsoft git repo - since the head version (245.2) has this:

ARG VARIANT=bullseye
FROM mcr.microsoft.com/vscode/devcontainers/base:${VARIANT}

I could try removing the "0-" and/or changing the value for the VARIANT - but does the value from the devcontainer.json actually override that value - i.e. if we have the following in our devcontainer.json?

	"build": {
		"dockerfile": "Dockerfile",
		// Update 'VARIANT' to pick an Debian version: bullseye, buster
		// Use bullseye on local arm64/Apple Silicon.
		"args": { "VARIANT": "bullseye" }
	},

[samruddhikhandale]

Reading https://github.com/devcontainers/cli/issues/349 looks like docker credential helpers are not supported for Features yet. I believe that should explain why they aren't working now but used to earlier.

[ghost]

Sorry I just realized that in your suggestion - you mentioned the following:

"enableNonRootDocker": "true",

which doesn't look like it is a default value. I'll give that a try - thanks!

[ghost]

No luck - same error -

i.e. even if I change the config to this:

    "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
      "version": "latest",
      "enableNonRootDocker": "true",
      "moby": "true"
    },

python fails to run the following command:

subprocess.CalledProcessError: Command '['/usr/local/bin/docker-credential-dev-containers-<hash number>, 'get']' returned non-zero exit status 255.

[samruddhikhandale]

Reading https://github.com/devcontainers/cli/issues/349 looks like docker credential helpers are not supported for Features yet. I believe that should explain why they aren't working now but used to earlier.

Looks like you've missed https://github.com/devcontainers/features/issues/376#issuecomment-1372862313

Does that make sense to you?

[ghost]

I saw it - but I had hoped that your earlier comment (with the recommendation to update my config) would resolve it.

So for now - it's just "unsupported" - and I should just keep an eye on this issue: https://github.com/devcontainers/cli/issues/349

and in the meantime - just continue to blow away the contents of:

~/.docker/config.json

?

Thanks again!

[samruddhikhandale]

Looks relevant to https://github.com/microsoft/vscode-remote-release/issues/7982

@source-transformer can you retry with Dev Containers 0.279.0-pre-release version and see if it fixes the issue?

[NewGyu]

Hello, I also have the same problem when using "Feature". On my environment, it perfectly reproduces when "Feature" is given in .devcontainer.json.

[2024-03-20T02:37:15.235Z] [+] Building 2.6s (3/3) FINISHED                                 docker:default
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load build definition from Dockerfile.extended              0.0s
 => => transferring dockerfile: 2.04kB                                     0.0s
 => ERROR resolve image config for docker.io/docker/dockerfile:1.4         2.6s
------
 > resolve image config for docker.io/docker/dockerfile:1.4:
------
[2024-03-20T02:37:15.236Z] Dockerfile.extended:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1.4
   2 |     ARG _DEV_CONTAINERS_BASE_IMAGE=placeholder
   3 |     
--------------------
ERROR: failed to solve: error getting credentials - err: exit status 127, out: ``
[2024-03-20T02:37:15.242Z] Stop (6374 ms): Run: docker buildx build --load --build-context dev_containers_feature_content_source=/tmp/devcontainercli-newgyu/container-features/0.56.2-1710902227415 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:bullseye --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-tmp-d34e0e6fb857daf91a38d46894e3c11f45f87b023664e43e4cc4856c0961216e-features -f /tmp/devcontainercli-newgyu/container-features/0.56.2-1710902227415/Dockerfile.extended /tmp/devcontainercli-newgyu/empty-folder
[2024-03-20T02:37:15.243Z] Error: Command failed: docker buildx build --load --build-context dev_containers_feature_content_source=/tmp/devcontainercli-newgyu/container-features/0.56.2-1710902227415 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:bullseye --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-tmp-d34e0e6fb857daf91a38d46894e3c11f45f87b023664e43e4cc4856c0961216e-features -f /tmp/devcontainercli-newgyu/container-features/0.56.2-1710902227415/Dockerfile.extended /tmp/devcontainercli-newgyu/empty-folder
[2024-03-20T02:37:15.243Z]     at atA (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:464:1260)
[2024-03-20T02:37:15.243Z]     at yH (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:464:1002)
[2024-03-20T02:37:15.243Z]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-03-20T02:37:15.243Z]     at async StA (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:481:3660)
[2024-03-20T02:37:15.243Z]     at async ZC (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:481:4775)
[2024-03-20T02:37:15.243Z]     at async trA (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:614:11269)
[2024-03-20T02:37:15.243Z]     at async erA (/home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js:614:11010)
[2024-03-20T02:37:15.247Z] Stop (13155 ms): Run in Host: /home/newgyu/.vscode-remote-containers/bin/863d2581ecda6849923a2118d93a088b0745d9d6/node /home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js up --container-session-data-folder /tmp/devcontainers-a815409a-0d06-470e-8fed-7aed1e811d8d1710902215445 --workspace-folder /home/newgyu/tmp --workspace-mount-consistency cached --id-label devcontainer.local_folder=\\wsl.localhost\Ubuntu-22.04\home\newgyu\tmp --id-label devcontainer.config_file=/home/newgyu/tmp/.devcontainer/devcontainer.json --log-level debug --log-format json --config /home/newgyu/tmp/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --mount type=bind,source=/run/user/1000/wayland-0,target=/tmp/vscode-wayland-510b3d34-bd44-4ecf-80b1-ccaa0c3397e7.sock --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root
[2024-03-20T02:37:15.247Z] Exit code 1
[2024-03-20T02:37:15.250Z] Command failed: /home/newgyu/.vscode-remote-containers/bin/863d2581ecda6849923a2118d93a088b0745d9d6/node /home/newgyu/.vscode-remote-containers/dist/dev-containers-cli-0.348.0/dist/spec-node/devContainersSpecCLI.js up --container-session-data-folder /tmp/devcontainers-a815409a-0d06-470e-8fed-7aed1e811d8d1710902215445 --workspace-folder /home/newgyu/tmp --workspace-mount-consistency cached --id-label devcontainer.local_folder=\\wsl.localhost\Ubuntu-22.04\home\newgyu\tmp --id-label devcontainer.config_file=/home/newgyu/tmp/.devcontainer/devcontainer.json --log-level debug --log-format json --config /home/newgyu/tmp/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --mount type=bind,source=/run/user/1000/wayland-0,target=/tmp/vscode-wayland-510b3d34-bd44-4ecf-80b1-ccaa0c3397e7.sock --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root
[2024-03-20T02:37:15.251Z] Exit code 1
[2024-03-20T02:38:53.720Z] Start: Run: wsl -d Ubuntu-22.04 -e wslpath -u \\wsl.localhost\Ubuntu-22.04\home\newgyu\tmp
[2024-03-20T02:38:53.868Z] Stop (148 ms): Run: wsl -d Ubuntu-22.04 -e wslpath -u \\wsl.localhost\Ubuntu-22.04\home\newgyu\tmp

$ code --version 1.87.2 863d2581ecda6849923a2118d93a088b0745d9d6 x64

c:>code --list-extensions --show-versions [email protected] [email protected]

Note: @0.352.0 did not change the result.

$ docker version Client: Version: 24.0.7-rd API version: 1.42 (downgraded from 1.43) Go version: go1.20.10 Git commit: 72ffacf Built: Wed Nov 1 18:42:47 2023 OS/Arch: linux/amd64 Context: default

Server: Engine: Version: 23.0.6 API version: 1.42 (minimum version 1.12) Go version: go1.20.11 Git commit: 9dbdbd4b6d7681bd18c897a6ba0376073c2a72ff Built: Fri Nov 17 20:59:57 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.2 GitCommit: 0cae528dd6cb557f7201036e9f43420650207b58 runc: Version: 1.1.12 GitCommit: 51d5e94601ceffbbd85688df1c928ecccbfa4685 docker-init: Version: 0.19.0 GitCommit:

$ uname -a Linux GALLERIA 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

systeminfo ホスト名: GALLERIA OS 名: Microsoft Windows 11 Home OS バージョン: 10.0.22631 N/A ビルド 22631 OS 製造元: Microsoft Corporation

example case 1

{
	"name": "Debian",
	"image": "mcr.microsoft.com/devcontainers/base:bullseye",
	"features": {
		"ghcr.io/devcontainers/features/dotnet:2": {}
	}
}

example case 2

{
	"name": "Node.js & TypeScript",
	"image": "mcr.microsoft.com/devcontainers/typescript-node:1-20-bullseye",
	"features": {
		"ghcr.io/devcontainers/features/python:1": {
			"version": "3.11"
		}
	}
}

One thing I do have in mind is that my docker environment was changed from Docker Deskto to Rancher Desktop a few monthes ago. I'm not sure that causes this problem or not.

[NewGyu]

Hello, changing credsStore value to "pass" resolves this.

$ cat ~/.docker/config.json
{
  "credsStore": "pass"
}

In addition, my environment could not resolve the name of ghcr.io. In order to fix this, it was necessary to change nameserver to 8.8.8.8.

$ cat /etc/resolv.conf
nameserver 8.8.8.8

Note: By default, resolv.conf will be automatically restored by WSL2, therefore it is necessary to set generateResolveConf = false.

$ cat /etc/wsl.conf

[boot]
systemd=true

[network]
generateResolvConf = false