Registry authentication with credential stores/helpers is not available while cloning a repo to a volume

[davidwallacejackson]

The bootstrap container that VS Code uses to clone a repository for the Clone Repository in Container Volume feature mounts the host's docker config, but can't access configured credential stores or helpers from the host. As a result, the container can't access images that are stored in private registries unless registry authentication is configured using a plain login and no credential store, which makes it challenging to use with services that rely on credential helpers (e.g. Google Container/Artifact Registry) or even with Docker Hub if it was configured through Docker Desktop (which, at least on Mac, uses the OS' credential store).

• VSCode Version: 1.71.0
• Local OS Version: MacOS 12.3.1
• Remote OS Version:
• Remote Extension/Connection Type: SSH/Containers/WSL/Server
• Logs:

[20 ms] Remote-Containers 0.251.0 in VS Code 1.71.0 (784b0177c56c607789f9638da7b6bf3230d47a8c).
[20 ms] Start: Resolving Remote
[25 ms] Start: Check Docker is running
[25 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[102 ms] Server API version: 1.41
[103 ms] Start: Run: docker volume ls -q
[168 ms] Start: Run: docker volume create --label vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --label vsch.local.repository.unique=false --label vsch.local.repository.branch=master dc-test4
[248 ms] Start: Run: docker build -f /var/folders/wk/pbzbnm1j55nb32_8v0dvyr3w0000gn/T/vsch/bootstrap-image/0.251.0/bootstrap.Dockerfile -t vsc-volume-bootstrap /var/folders/wk/pbzbnm1j55nb32_8v0dvyr3w0000gn/T/vsch/bootstrap-image/0.251.0
[+] Building 0.3s (9/9) FINISHED
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 47B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/b  0.2s
 => [1/4] FROM mcr.microsoft.com/vscode/devcontainers/base:0-alpine-3.14@  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 5.90kB                                        0.0s
 => CACHED [2/4] RUN apk add --no-cache  nodejs  npm  make  g++  docker-c  0.0s
 => CACHED [3/4] RUN cd && npm i node-pty                                  0.0s
 => CACHED [4/4] COPY .vscode-remote-containers /root/.vscode-remote-cont  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:f003838d314628308758a3d7cd75b5bd4bd56f66131d1  0.0s
 => => naming to docker.io/library/vsc-volume-bootstrap                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[892 ms] Cloning Github repository: davidwallacejackson/devcontainer-pull-test into /workspaces/devcontainer-pull-test

[892 ms] Start: Run: docker run -d --mount type=volume,src=dc-test4,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[1146 ms] Start: Run in container: /bin/sh
[1148 ms] Start: Launching Remote-Containers helper.
[1149 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-accb9ab985d23e06bdce0537063babc4990d4361.sock) forwarded to local host (/private/tmp/com.apple.launchd.4Cx3DcdyvD/Listeners).
[1149 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[1151 ms] Start: Run in container: /bin/sh
[1154 ms] Start: Run in container: echo ~
[1274 ms] /root
[1274 ms]
[1274 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-accb9ab985d23e06bdce0537063babc4990d4361.js' >/tmp/vscode-remote-containers-accb9ab985d23e06bdce0537063babc4990d4361.js
[1276 ms]
[1276 ms]
[1277 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-accb9ab985d23e06bdce0537063babc4990d4361.js' >/tmp/vscode-remote-containers-server-accb9ab985d23e06bdce0537063babc4990d4361.js_1663053425430
[1288 ms]
[1289 ms]
[1290 ms] Start: Run in container: # Test for /root/.gitconfig and git
[1292 ms]
[1292 ms]
[1292 ms] Start: Run in container: # Copy /Users/davidjackson/.gitconfig to /root/.gitconfig
[1295 ms]
[1295 ms]
[1296 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-accb9ab985d23e06bdce0537063babc4990d4361.js $*; }; f' || true
[1298 ms]
[1298 ms]
[1299 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[1300 ms]
[1300 ms]
[1300 ms] Start: Run in container: # Copy /Users/davidjackson/.ssh/known_hosts to /root/.ssh/known_hosts
[1318 ms]
[1318 ms]
[1355 ms] Start: Run in container: git clone --depth 1 --branch master https://github.com/davidwallacejackson/devcontainer-pull-test.git .
Cloning into '.'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 0), reused 3 (delta 0), pack-reused 0
Receiving objects: 100% (6/6), 4.33 KiB | 4.33 MiB/s, done.
[2516 ms] Start: Run in container: cat /workspaces/devcontainer-pull-test/.devcontainer/devcontainer.json 2>/dev/null
[2519 ms] Start: Run in container: cat /workspaces/devcontainer-pull-test/.devcontainer/devcontainer.json 2>/dev/null
[2523 ms] Start: Run in container: cat <<'EOF-/tmp/devcontainer-1c2e34ca-d9f8-49e3-aaa0-4d5a5fa9d247.json' >/tmp/devcontainer-1c2e34ca-d9f8-49e3-aaa0-4d5a5fa9d247.json_1663053426677
[2525 ms]
[2525 ms]
[2525 ms] Start: Run in container: docker volume ls -q
[2971 ms] Start: Run in container: docker ps -q -a --filter label=vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --filter label=vsch.local.repository.volume=dc-test4 --filter label=vsch.local.repository.folder=devcontainer-pull-test --filter label=vsch.quality=stable
[3093 ms] Start: Run in container: docker ps -q -a --filter label=vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --filter label=vsch.local.repository.volume=dc-test4 --filter label=vsch.local.repository.folder=devcontainer-pull-test
[3419 ms] Start: Run in container: node /root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js up --workspace-folder /workspaces/devcontainer-pull-test --workspace-mount-consistency cached --id-label vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --id-label vsch.local.repository.volume=dc-test4 --id-label vsch.local.repository.folder=devcontainer-pull-test --log-level debug --log-format json --config /workspaces/devcontainer-pull-test/.devcontainer/devcontainer.json --override-config /tmp/devcontainer-1c2e34ca-d9f8-49e3-aaa0-4d5a5fa9d247.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=dc-test4,target=/workspaces,external=true --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default off --mount-workspace-git-root true --terminal-columns 119 --terminal-rows 28
[3612 ms] remote-containers 0.251.0.
[3612 ms] Start: Run: docker buildx version
[3642 ms] Start: Resolving Remote
[3644 ms] Start: Run: git rev-parse --show-cdup
[3648 ms] Start: Run: docker ps -q -a --filter label=vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --filter label=vsch.local.repository.volume=dc-test4 --filter label=vsch.local.repository.folder=devcontainer-pull-test
[3682 ms] local container features stored at: /root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/node_modules/vscode-dev-containers/container-features
[3683 ms] Start: Run: tar --no-same-owner -x -f -
[3704 ms] Start: Run: docker -v
[3728 ms] Start: Run: docker events --format {{json .}} --filter event=start
[3730 ms] Start: Starting container
[3731 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=volume,src=dc-test4,dst=/workspaces --mount type=volume,src=vscode,dst=/vscode -l vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master -l vsch.local.repository.volume=dc-test4 -l vsch.local.repository.folder=devcontainer-pull-test --entrypoint /bin/sh davidwandb/devcontainer-pull-test -c echo Container started
Unable to find image 'davidwandb/devcontainer-pull-test:latest' locally
[2452 ms] Start: Run in container: git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
[2619 ms] Start: Run in container: git fetch --unshallow --tags
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 8 (delta 4), reused 4 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), 680 bytes | 680.00 KiB/s, done.
docker: Error response from daemon: pull access denied for davidwandb/devcontainer-pull-test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
[5193 ms] Start: Run: docker ps -q -a --filter label=vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --filter label=vsch.local.repository.volume=dc-test4 --filter label=vsch.local.repository.folder=devcontainer-pull-test
[5224 ms] Error: Command failed: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=volume,src=dc-test4,dst=/workspaces --mount type=volume,src=vscode,dst=/vscode -l vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master -l vsch.local.repository.volume=dc-test4 -l vsch.local.repository.folder=devcontainer-pull-test --entrypoint /bin/sh davidwandb/devcontainer-pull-test -c echo Container started
[5224 ms] trap "exit 0" 15
[5224 ms] exec "$@"
[5224 ms] while sleep 1 & wait $!; do :; done -
[5224 ms]     at SF (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:2186)
[5224 ms]     at pb (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:2124)
[5224 ms]     at processTicksAndRejections (internal/process/task_queues.js:95:5)
[5224 ms]     at async DF (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:2006)
[5224 ms]     at async to (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:3110)
[5224 ms]     at async Ak (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:383:8108)
[5224 ms]     at async Ok (/root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js:383:7864)
[5235 ms] Exit code 1
[5235 ms] Start: Run: docker rm -f 421e310db8263700460d2424196fddc5ad7fa72be97c57f3dbbbf176da5acdee
[5241 ms] Command failed: node /root/.vscode-remote-containers/dist/dev-containers-cli-0.251.0/dist/spec-node/devContainersSpecCLI.js up --workspace-folder /workspaces/devcontainer-pull-test --workspace-mount-consistency cached --id-label vsch.local.repository=https://github.com/davidwallacejackson/devcontainer-pull-test.git/tree/master --id-label vsch.local.repository.volume=dc-test4 --id-label vsch.local.repository.folder=devcontainer-pull-test --log-level debug --log-format json --config /workspaces/devcontainer-pull-test/.devcontainer/devcontainer.json --override-config /tmp/devcontainer-1c2e34ca-d9f8-49e3-aaa0-4d5a5fa9d247.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=dc-test4,target=/workspaces,external=true --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default off --mount-workspace-git-root true --terminal-columns 119 --terminal-rows 28
[5241 ms] Exit code 1
[5313 ms] Container server terminated (code: 137, signal: null).

Steps to Reproduce: see repro here

Does this issue occur when you try this locally?: Yes Does this issue occur when you try this locally and all extensions are disabled?: N/A (this is a problem with an extension)

[Chuxel]

@chrmarti @bamurtaugh This originated from a Twitter conversation for reference (https://twitter.com/ForSpareParts/status/1569746805797322752) and affects the "continue on" flow as well. To resolve, we'd likely need to inspect docker config and if credsStore is set... or perhaps if we pull on the client side if the image isn't already available locally first, the inner docker would be able to access it (assuming it doesn't also try to pull).

[bamurtaugh]

Thanks for flagging @Chuxel. Just to ensure I understand, what do we mean by "Continue on" in this case?

[Chuxel]

@bamurtaugh This relates to the continue on experience in vscode.dev that was designed to open up in a container volume. The "Remote - Containers | Open" badge is the same thing. If the repo just has a Dockerfile that references a public image (or just refs a public image directly), there's no issue - its pre-built private images that require a docker sign in to access. Given the recommendation to pre-build images for perf (and how easy it is to do with GHCR and Actions), that's problematic.

So, it affects Remote - Containers "Clone Repository in Container Volume" locally, vscode.dev => Remote - Containers, and README badge => Remote - Containers.

I'm not sure if we originally pulled the image first on the host which avoided this and some refactoring caused a regression, or if this is a gap.

[davidwallacejackson]

or perhaps if we pull on the client side if the image isn't already available locally first, the inner docker would be able to access it (assuming it doesn't also try to pull).

Nitpick -- it'd be great if we had the ability to always pull before bootstrapping, even if the image is already available. I build my devcontainer image in CI, and ideally users would just get the newest thing by default whenever they rebuild.

The way I'm planning to handle this right now is to have the devcontainer itself pull the newest version of its own image upon startup, such that the next rebuild will always be up-to-date, but being able to pull at the time of container creation would be even better.