vscode-remote-release
vscode-remote-release copied to clipboard
microsoft
Building New Dev Containers Fails with WSL/podman
I have used VSCode with Windows 10/WSL2 to build lots of dev containers. I haven't had to build any recently until someone pointed out that it's no longer working. So I cloned an existing project that I currently have a dev container operating, and VSCode will not build a new dev container for the new repository.
I mean any dev container I try and build will fail so it should be pretty easy to reproduce.
I recall a long time ago that the whole "permission" thing had something to do with the user inside the container but that was for different containers that required being ROOT inside them or something. I have never had this issue until recently with podman.
Version: 1.100.3 (user setup) Commit: 258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3 Date: 2025-06-02T13:30:54.273Z Electron: 34.5.1 ElectronBuildId: 11369351 Chromium: 132.0.6834.210 Node.js: 20.19.0 V8: 13.2.152.41-electron.0 OS: Windows_NT x64 10.0.19045
[37 ms] Dev Containers 0.413.0 in VS Code 1.100.3 (258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3).
[37 ms] Start: Run: wsl -d Ubuntu-24.04 -e wslpath -u \\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder
[560 ms] Start: Resolving Remote
[579 ms] Start: Run: wsl -d Ubuntu-24.04 -e wslpath -u \\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder
[1002 ms] Start: Run: wsl -d Ubuntu-24.04 -e /bin/sh -c cd '/home/bramankp/nano-events-transcoder' && /bin/sh
[1022 ms] Start: Run in host: id -un
[1240 ms] bramankp
[1240 ms]
[1240 ms] Start: Run in host: (command -v getent >/dev/null 2>&1 && getent passwd 'bramankp' || grep -E '^bramankp|^[^:]*:[^:]*:bramankp:' /etc/passwd || true)
[1243 ms] Start: Run in host: echo ~
[1244 ms] /home/bramankp
[1244 ms]
[1244 ms] Start: Run in host: test -f '/home/bramankp/.vscode-server/cli/servers/Stable-258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3/server/node'
[1245 ms]
[1245 ms]
[1245 ms] Exit code 1
[1246 ms] Start: Run in host: test -f '/home/bramankp/.vscode/cli/servers/Stable-258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3/server/node'
[1260 ms]
[1260 ms]
[1260 ms] Exit code 1
[1261 ms] Start: Run in host: test -f '/home/bramankp/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3/node'
[1264 ms]
[1266 ms]
[1266 ms] Start: Run in host: test -f '/home/bramankp/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3/node_modules/node-pty/package.json'
[1267 ms]
[1267 ms]
[1267 ms] Start: Run in host: test -f '/home/bramankp/.vscode-remote-containers/dist/vscode-remote-containers-server-0.413.0.js'
[1284 ms]
[1284 ms]
[1287 ms] userEnvProbe: loginInteractiveShell (default)
[1288 ms] userEnvProbe: not found in cache
[1288 ms] userEnvProbe shell: /bin/bash
[1449 ms] userEnvProbe PATHs:
Probe: '/home/bramankp/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Python313/Scripts/:/mnt/c/Python313/:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/WindowsPowerShell/Scripts:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/PuTTY/:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/CMake/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/LLVM/bin:/mnt/c/Program Files (x86)/LLVM/bin:/mnt/c/Program Files/RedHat/Podman/:/mnt/c/Program Files/Nordic Semiconductor/nrf-command-line-tools/bin/:/mnt/c/Users/pabraman2001/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/pabraman2001/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/pabraman2001/AppData/Local/Programs/qm/bin:/mnt/c/Users/pabraman2001/.dotnet/tools:/mnt/c/ProgramData/mingw64/mingw64/bin:/snap/bin:/home/bramankp/.local/bin'
Container: None
[1451 ms] Setting up container for folder or workspace: /home/bramankp/nano-events-transcoder
[1496 ms] Start: Check Docker is running
[1496 ms] Start: Run in Host: podman version
[2094 ms] Client: Podman Engine
Version: 4.9.3
API Version: 4.9.3
Go Version: go1.22.2
Built: Wed Dec 31 19:00:00 1969
OS/Arch: linux/amd64
[2097 ms] Start: Run in Host: podman volume ls -q
[2455 ms] Start: Run in Host: podman ps -q -a --filter label=vsch.local.folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder --filter label=vsch.quality=stable
[2924 ms] Start: Run in Host: podman ps -q -a --filter label=devcontainer.local_folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder --filter label=devcontainer.config_file=/home/bramankp/nano-events-transcoder/.devcontainer.json
[3292 ms] Start: Run in Host: podman inspect --type container 53ad233a23e0
[3752 ms] Start: Run in Host: podman ps -q -a --filter label=devcontainer.local_folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder
[4143 ms] Start: Run in Host: podman inspect --type container 53ad233a23e0
[4565 ms] Start: Run in Host: podman exec -i -u root 53ad233a23e05ce90f5c42bddc0ff5757cb5057000e8276a9f9dd1f5a6c4a6fc /bin/sh -c echo "Container already running. Keep-alive process started." ; export VSCODE_REMOTE_CONTAINERS_SESSION=24051198-52a5-44ee-aa50-e1f8108c5b6a1749731725507 ; /bin/sh
[4566 ms] Running Dev Containers CLI: up --docker-path podman --container-session-data-folder /tmp/devcontainers-24051198-52a5-44ee-aa50-e1f8108c5b6a1749731725507 --workspace-folder /home/bramankp/nano-events-transcoder --workspace-mount-consistency cached --gpu-availability detect --id-label devcontainer.local_folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder --id-label devcontainer.config_file=/home/bramankp/nano-events-transcoder/.devcontainer.json --log-level debug --log-format json --config /home/bramankp/nano-events-transcoder/.devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --mount type=bind,source=/run/user/1002/wayland-0,target=/tmp/vscode-wayland-5239a58c-1b8a-4d89-8237-c4a303db6572.sock --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --terminal-columns 308 --terminal-rows 13 --include-configuration --include-merged-configuration
[4566 ms] Start: Checking for Dev Containers CLI
[4576 ms] Start: Run in Host: /home/bramankp/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3/node /home/bramankp/.vscode-remote-containers/dist/dev-containers-cli-0.413.0/dist/spec-node/devContainersSpecCLI.js up --docker-path podman --container-session-data-folder /tmp/devcontainers-24051198-52a5-44ee-aa50-e1f8108c5b6a1749731725507 --workspace-folder /home/bramankp/nano-events-transcoder --workspace-mount-consistency cached --gpu-availability detect --id-label devcontainer.local_folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder --id-label devcontainer.config_file=/home/bramankp/nano-events-transcoder/.devcontainer.json --log-level debug --log-format json --config /home/bramankp/nano-events-transcoder/.devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --mount type=bind,source=/run/user/1002/wayland-0,target=/tmp/vscode-wayland-5239a58c-1b8a-4d89-8237-c4a303db6572.sock --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --terminal-columns 308 --terminal-rows 13 --include-configuration --include-merged-configuration
[4817 ms] @devcontainers/cli 0.76.0. Node.js v20.19.0. linux 6.6.87.1-microsoft-standard-WSL2 x64.
[4817 ms] Start: Run: podman buildx version
[5036 ms] Container already running. Keep-alive process started.
[5530 ms] buildah 1.33.7
[5530 ms]
[5530 ms] Start: Run: podman -v
[5553 ms] Start: Resolving Remote
[5557 ms] Start: Run: git rev-parse --show-cdup
[5564 ms] Start: Run: podman ps -q -a --filter label=devcontainer.local_folder=\\wsl.localhost\Ubuntu-24.04\home\bramankp\nano-events-transcoder --filter label=devcontainer.config_file=/home/bramankp/nano-events-transcoder/.devcontainer.json
[5978 ms] Start: Run: podman inspect --type container 53ad233a23e0
[6380 ms] Start: Inspecting container
[6380 ms] Start: Run: podman inspect --type container 53ad233a23e05ce90f5c42bddc0ff5757cb5057000e8276a9f9dd1f5a6c4a6fc
[6772 ms] Start: Run in container: /bin/sh
[6775 ms] Start: Run in container: uname -m
[7266 ms] x86_64
[7266 ms]
[7266 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[7273 ms] PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
[7273 ms]
[7273 ms] Start: Run in container: (command -v getent >/dev/null 2>&1 && getent passwd '1002' || grep -E '^1002|^[^:]*:[^:]*:1002:' /etc/passwd || true)
[7290 ms] Start: Run in container: test -f '/var/devcontainer/.patchEtcEnvironmentMarker'
[7292 ms]
[7292 ms]
[7292 ms] Exit code 1
[7292 ms] Start: Run in container: /bin/sh
[7294 ms] Start: Run in container: test ! -f '/var/devcontainer/.patchEtcEnvironmentMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcEnvironmentMarker' ; } 2> /dev/null
[7766 ms]
[7766 ms]
[7766 ms] Start: Run in container: cat >> /etc/environment <<'etcEnvrionmentEOF'
[7770 ms]
[7770 ms]
[7771 ms] Start: Run in container: test -f '/var/devcontainer/.patchEtcProfileMarker'
[7772 ms]
[7772 ms]
[7772 ms] Exit code 1
[7772 ms] Start: Run in container: test ! -f '/var/devcontainer/.patchEtcProfileMarker' && set -o noclobber && mkdir -p '/var/devcontainer' && { > '/var/devcontainer/.patchEtcProfileMarker' ; } 2> /dev/null
[7774 ms]
[7774 ms]
[7774 ms] Start: Run in container: sed -i -E 's/((^|\s)PATH=)([^\$]*)$/\1${PATH:-\3}/g' /etc/profile || true
[7781 ms]
[7781 ms]
[7756 ms] Start: Run in Host: podman inspect --type container 53ad233a23e05ce90f5c42bddc0ff5757cb5057000e8276a9f9dd1f5a6c4a6fc
[8062 ms] Start: Inspecting container
[8062 ms] Start: Run in Host: podman inspect --type container 53ad233a23e05ce90f5c42bddc0ff5757cb5057000e8276a9f9dd1f5a6c4a6fc
[8484 ms] Start: Run in Host: podman exec -i -u 1002:1002 -e VSCODE_REMOTE_CONTAINERS_SESSION=24051198-52a5-44ee-aa50-e1f8108c5b6a1749731725507 53ad233a23e05ce90f5c42bddc0ff5757cb5057000e8276a9f9dd1f5a6c4a6fc /bin/sh
[8486 ms] Start: Run in container: uname -m
[8995 ms] x86_64
[8995 ms]
[8996 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[8998 ms] PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
[8998 ms]
[8998 ms] Start: Run in container: (command -v getent >/dev/null 2>&1 && getent passwd '1002' || grep -E '^1002|^[^:]*:[^:]*:1002:' /etc/passwd || true)
[9001 ms] Start: Setup shutdown monitor
[9002 ms] Forking shutdown monitor: c:\Users\pabraman2001\.vscode\extensions\ms-vscode-remote.remote-containers-0.413.0\dist\shutdown\shutdownMonitorProcess \\.\pipe\vscode-remote-containers-87614302-da08-4559-b568-2f9c33b122d8-sock singleContainer Debug c:\Users\pabraman2001\AppData\Roaming\Code\logs\20250609T091228\window8\exthost\ms-vscode-remote.remote-containers 1749731726956
[9027 ms] Start: Run in container: test -d '/.vscode-server'
[9029 ms]
[9029 ms]
[9030 ms] Exit code 1
[9030 ms] Start: Run in container: test -d '/.vscode-remote'
[9031 ms]
[9031 ms]
[9031 ms] Exit code 1
[9032 ms] Start: Run in container: test ! -f '/.vscode-server/data/Machine/.writeMachineSettingsMarker' && set -o noclobber && mkdir -p '/.vscode-server/data/Machine' && { > '/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[9035 ms]
[9035 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[9036 ms] Exit code 1
[9036 ms] Start: Run in container: cat /.vscode-server/data/Machine/settings.json
[9039 ms]
[9039 ms] cat: /.vscode-server/data/Machine/settings.json: No such file or directory
[9039 ms] Exit code 1
[9040 ms] Start: Run in container: test -d '/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3'
[9042 ms]
[9042 ms]
[9043 ms] Exit code 1
[9043 ms] Start: Run in container: test -d '/vscode/vscode-server/bin/linux-x64/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3'
[9044 ms]
[9045 ms]
[9045 ms] Start: Run in container: mkdir -p '/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3' '/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3'
[9048 ms]
[9048 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[9048 ms] Exit code 1
[9052 ms] Command in container failed: mkdir -p '/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3' '/.vscode-server/bin/258e40fedc6cb8edf399a463ce3a9d32e7e1f6f3'
[9052 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[9053 ms] Exit code 1
There is a fix in the latest release (Dev Containers 0.417.0) that might address this. Could you give that a try?
Currently using 0.417.0 of that extension (and updated to 1.101.0 of VSC). No change in behavior.
I also did just try
Dev Containers 0.418.0 in VS Code 1.101.0 (dfaf44141ea9deb3b4096f7cd6d24e00c147a4b1).
with the same results.
We now automatically add --userns=keep-id to podman run unless we detect that VS Code will connect as root to the container. It looks like that detection might not work in your case. Could you share a minimal devcontainer.json and Dockerfile that demonstrate the problem with 0.417.0?
There's definitely something funky going on. I created a minimal test project and the same dev container builds and runs just fine. When I try and build the dev container in the primary project (same .devcontainer.json) it fails. Let me compare the log outputs and see if I can find the difference.
devcon-failure.txt devcon-success.txt
Both seem to start the same way. At about 4100ms they both run Running Dev Containers CLI: up --docker-path podman --container-session-data-folder using "user" 1002 in the command path.
At around 8400ms, the one that succeeds runs getent passwd with user "root" while the one that fails runs it with user "1002".
I simplified .devcontainer.json to just be "image":"ubuntu:24.04" as a final test and I get the same results. So it's something about the containing project or something that is the difference? But that doesn't make any sense to me.
Interesting ...
Just out of curiosity I went and removed all instances of previous dev containers using that image and then re-opened the project (which triggered rebuilding the container again).
And now it worked. I am wondering if there is something about having existing instances of that dev container that causes the difference.
Just tested it with someone else (who originally reported the problem) and clearing out all the old dev containers using that image allows all new ones to build successfully.
So I guess that's the solution to whatever weird problem this was. I will close the thread but feel free to wonder if that detection scheme broke somehow.