devpod icon indicating copy to clipboard operation
devpod copied to clipboard
verified publisher icon loft-sh

Unable to create VSCode sessions with K8s provider on Windows after upgrade

Open mkorejo opened this issue 1 year ago • 1 comments

What happened?
I upgraded my DevPod version from 0.2.x to latest version, 0.5.20. Following the upgrade, I can no longer start VSCode sessions with the Kubernetes provider. VSCode Browser sessions are working. VSCode launches, the remote SSH extension kicks in, the following output is displayed, and a dialog popup is displayed saying Could not connect:

[10:46:28.582] Log Level: 1
[10:46:28.606] VS Code version: 1.93.1
[10:46:28.607] Remote-SSH version: [email protected]
[10:46:28.607] win32 x64
[10:46:28.612] SSH Resolver called for "ssh-remote+murad.devpod", attempt 1
[10:46:28.613] "remote.SSH.useLocalServer": false
[10:46:28.613] "remote.SSH.useExecServer": true
[10:46:28.614] "remote.SSH.showLoginTerminal": false
[10:46:28.614] "remote.SSH.remotePlatform": {"*.devpod":"linux","10.210.98.152":"linux"}
[10:46:28.614] "remote.SSH.path": undefined
[10:46:28.614] "remote.SSH.configFile": undefined
[10:46:28.614] "remote.SSH.useFlock": true
[10:46:28.615] "remote.SSH.lockfilesInTmp": false
[10:46:28.615] "remote.SSH.localServerDownload": auto
[10:46:28.615] "remote.SSH.remoteServerListenOnSocket": false
[10:46:28.615] "remote.SSH.showLoginTerminal": false
[10:46:28.615] "remote.SSH.defaultExtensions": []
[10:46:28.616] "remote.SSH.loglevel": 1
[10:46:28.616] "remote.SSH.enableDynamicForwarding": true
[10:46:28.616] "remote.SSH.enableRemoteCommand": false
[10:46:28.616] "remote.SSH.serverPickPortsFromRange": {}
[10:46:28.616] "remote.SSH.serverInstallPath": {}
[10:46:28.616] "remote.SSH.permitPtyAllocation": false
[10:46:28.617] "remote.SSH.preferredLocalPortRange: undefined
[10:46:28.617] "remote.SSH.useCurlAndWgetConfigurationFiles: false
[10:46:28.638] SSH Resolver called for host: murad.devpod
[10:46:28.638] Setting up SSH remote "murad.devpod"
[10:46:28.646] Using commit id "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40" and quality "stable" for server
[10:46:28.652] Script variables:
 {
  "InstallExitCode.24": "AlreadyInProgress",
  "InstallExitCode.25": "ServerDownloadFailed",
  "InstallExitCode.26": "NoDownloaderAvailable",
  "InstallExitCode.27": "UnsupportedArch",
  "InstallExitCode.28": "StatusCheckFailed",
  "InstallExitCode.29": "NeedInsidersArch",
  "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck",
  "InstallExitCode.31": "ServerTransferFailed",
  "InstallExitCode.32": "ServerFailedToStart",
  "InstallExitCode.33": "NeedInsidersWindows",
  "InstallExitCode.34": "CreateInstallDirFailed",
  "InstallExitCode.35": "UnsupportedPlatform",
  "InstallExitCode.36": "ServerTerminatedCVE20201416",
  "InstallExitCode.37": "UnpackFailed",
  "InstallExitCode.38": "ChangeDirFailed",
  "InstallExitCode.AlreadyInProgress": "24",
  "InstallExitCode.ServerDownloadFailed": "25",
  "InstallExitCode.NoDownloaderAvailable": "26",
  "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30",
  "InstallExitCode.UnsupportedArch": "27",
  "InstallExitCode.StatusCheckFailed": "28",
  "InstallExitCode.NeedInsidersArch": "29",
  "InstallExitCode.ServerTransferFailed": "31",
  "InstallExitCode.ServerFailedToStart": "32",
  "InstallExitCode.NeedInsidersWindows": "33",
  "InstallExitCode.CreateInstallDirFailed": "34",
  "InstallExitCode.UnsupportedPlatform": "35",
  "InstallExitCode.ServerTerminatedCVE20201416": "36",
  "InstallExitCode.UnpackFailed": "37",
  "InstallExitCode.ChangeDirFailed": "38",
  "InstallUnpackCode.Success": "success",
  "InstallUnpackCode.Error": "error",
  "InstallUnpackCode.MissingFiles": "missingFiles",
  "uuid": "e7fac4deac8e",
  "startMarker": "e7fac4deac8e: running",
  "commitId": "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40",
  "quality": "stable",
  "token": "111aa111-11a1-1111-11aa-a11aa1a1a1aa",
  "vscodeAgentFolder": "$HOME/.vscode-server",
  "allowClientDownload": "1",
  "forceClientDownload": "0",
  "cliNameInArchive": "code",
  "ignoreWgetConfigFlag": " --no-config ",
  "ignoreCurlConfigFlag": " --disable ",
  "wgetTriesSegment": "--tries=1",
  "listenArgs": "--on-host=127.0.0.1 --on-port",
  "getDownloadServerStartTrigger": "e7fac4deac8e:trigger_server_download",
  "getDownloadServerEndTrigger": "e7fac4deac8e:trigger_server_download_end",
  "getProgressDownloading": "e7fac4deac8e%%1%%",
  "getProgressInstalling": "e7fac4deac8e%%2%%"
}
[10:46:28.656] Install and start server if needed
[10:46:28.663] Checking ssh with "C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe -V"
[10:46:28.669] Got error from ssh: spawn C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe ENOENT
[10:46:28.670] Checking ssh with "C:\Windows\system32\ssh.exe -V"
[10:46:28.672] Got error from ssh: spawn C:\Windows\system32\ssh.exe ENOENT
[10:46:28.673] Checking ssh with "C:\Windows\ssh.exe -V"
[10:46:28.675] Got error from ssh: spawn C:\Windows\ssh.exe ENOENT
[10:46:28.676] Checking ssh with "C:\Windows\System32\Wbem\ssh.exe -V"
[10:46:28.680] Got error from ssh: spawn C:\Windows\System32\Wbem\ssh.exe ENOENT
[10:46:28.680] Checking ssh with "C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe -V"
[10:46:28.682] Got error from ssh: spawn C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT
[10:46:28.683] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[10:46:28.862] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

[10:46:28.878] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 62083 "murad.devpod" sh
[10:46:28.916] Generated SSH command: 'type "D:\Users\MKorejo\AppData\Local\Temp\1\vscode-linux-multi-line-command-murad.devpod-839016287.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 62083 "murad.devpod" sh'
[10:46:28.917] Terminal shell path: C:\Windows\System32\cmd.exe
[10:46:29.857] "install" terminal received data: "[0m[0K"
[10:46:29.858] Got some output, clearing connection timeout
[10:46:29.957] "install" terminal received data: "CreateProcessW failed error:87[0K[?25l
posix_spawn: Unknown error[0K
The process tried to write to a nonexistent pipe.
[0K[?25h"
[10:46:30.259] "install" terminal command done
[10:46:30.260] Install terminal quit with output: posix_spawn: Unknown error
[10:46:30.260] Received install output: posix_spawn: Unknown error
[10:46:30.260] Stopped parsing output early. Remaining text: posix_spawn: Unknown error
[10:46:30.261] Failed to parse remote port from server output
[10:46:30.266] Resolver error: Error: 
	at v.Create (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:501404)
	at t.handleInstallOutput (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:498726)
	at t.tryInstall (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:616460)
	at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:575788
	at async t.withShowDetailsEvent (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:579017)
	at async k (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:572574)
	at async t.resolve (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:576428)
	at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:844803
[10:46:30.282] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"","arch":"","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"0","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0","didLocalDownload":"0","installUnpackCode":"0","outcome":"failure","reason":"UnparsableOutput","exitCodeLabel":""},"measures":{"resolveAttempts":1,"isExecServer":0,"timing.totalResolveTime":1650,"timing.preSshTime":305,"timing.scriptTime":1343}}
[10:46:30.289] ------

What did you expect to happen instead?

VSCode opens and mounts my PVC and I can start editing/working in the container.

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
  "image": "public.ecr.aws/v7d9y2d2/devpod:234f16d1",
  "workspaceMount": "source=${localWorkspaceFolder},target=/root,type=bind",
  "workspaceFolder": "/root",
  "forwardPorts": [8080, 9898],
  "overrideCommand": false,
  "privileged": true
}

Local Environment:

  • DevPod Version: v0.5.20
  • Operating System: windows
  • ARCH of the OS: AMD64

DevPod Provider:

  • Kubernetes Provider:
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.13+e709aa5", GitCommit:"d2902897cdc913a1bf6bc548ce26f8a427ef9b76", GitTreeState:"clean", BuildDate:"2024-05-13T13:50:21Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

Anything else we need to know?
This was working prior to upgrading DevPod. I had to upgrade the provider following DevPod upgrade with devpod provider upgrade kubernetes. I've tried multiple re-installations of DevPod, Git, and VSCode, and also regenerated my SSH identity, however nothing seems to work.

I can start a remote SSH explorer session in VSCode (outside of DevPod) to another EC2 instance in my environment. VSCode Browser also works in DevPod for the same workspace which doesn't work in VSCode.

mkorejo avatar Sep 27 '24 15:09 mkorejo

Hey @mkorejo, thanks for reporting the issue. Is there anything else that changed since the upgrade? I wasn't able to reproduce the issue with a very basic workspace and VSCode on windows with the kubernetes provider.

Can you confirm the same behaviour also occurs if you start the workspace with https://github.com/loft-sh/devpod-example-go?

pascalbreuninger avatar Sep 28 '24 13:09 pascalbreuninger

Sorry for the delay, @pascalbreuninger. I can confirm I get the same behavior:

[09:01:15.138] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[09:01:15.469] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

[09:01:15.488] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63982 "devpod-example-go.devpod" sh
[09:01:15.538] Generated SSH command: 'type "D:\Users\MKorejo\AppData\Local\Temp\1\vscode-linux-multi-line-command-devpod-example-go.devpod-968841258.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63982 "devpod-example-go.devpod" sh'
[09:01:15.542] Terminal shell path: C:\Windows\System32\cmd.exe
[09:01:18.156] "install" terminal received data: "[0m[0K"
[09:01:18.157] Got some output, clearing connection timeout
[09:01:18.213] "install" terminal received data: "CreateProcessW failed error:87[0K[?25l
posix_spawn: Unknown error[0K
The process tried to write to a nonexistent pipe.
[0K[?25h"
[09:01:18.508] "install" terminal command done
[09:01:18.509] Install terminal quit with output: posix_spawn: Unknown error
[09:01:18.510] Received install output: posix_spawn: Unknown error
[09:01:18.510] Stopped parsing output early. Remaining text: posix_spawn: Unknown error
[09:01:18.511] Failed to parse remote port from server output
[09:01:18.514] Resolver error: Error: 
	at v.Create (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:501404)
	at t.handleInstallOutput (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:498726)
	at t.tryInstall (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:616460)
	at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:575788
	at async t.withShowDetailsEvent (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:579017)
	at async k (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:572574)
	at async t.resolve (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:576428)
	at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:844803
[09:01:18.529] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"","arch":"","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"0","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0","didLocalDownload":"0","installUnpackCode":"0","outcome":"failure","reason":"UnparsableOutput","exitCodeLabel":""},"measures":{"resolveAttempts":1,"isExecServer":0,"timing.totalResolveTime":3469,"timing.preSshTime":498,"timing.scriptTime":2969}}
[09:01:18.536] ------

mkorejo avatar Oct 04 '24 14:10 mkorejo

This must've been a client-side installation issue with VSCode. It's working now.

mkorejo avatar Oct 20 '24 20:10 mkorejo