Expected behavior: Configuration for ssh with PyCharm, etc, works on Mac and Linux. Does not work on Windows.

Current behavior: IntelliJ fails to connect through ssh (specifically on Windows)

Bug details:

• Teleport version: 10
• Recreation steps: Configure pycharm with remote operator on Windows
• Debug logs: none contain errors or information

@elijahsgh Thank you for the ticket. Are there any relevant logs you can share? We will try to repro on our side as well.

@elijahsgh I was able to get PyCharm to work on Windows 10 and Teleport 10 and 11. Here are a few things to check as they are not mentioned in our official guide:

1. Make sure that the ssh-agent is running and has Teleport keys (tsh logout and tsh login should add them if they are missing).
2. Make sure that ~/.ssh/config exists and has the correct configuration. You can generate this file by calling tsh config > ~/.ssh/config
3. This may depend on the ssh implementation, but I noticed that Windows implementation of SSH requires the OpenSSH config file to have Unix instead of Windows line endings. You can convert the file by calling dos2unix. This is something that we can probably integrate into tsh itself.
4. Make sure that PyCharm is using OpenSSH as a config parser instead of the Legacy https://www.jetbrains.com/help/idea/advanced-settings.html#advanced-ssh

If you still have issues with it, please attach PyCharm logs https://intellij-support.jetbrains.com/hc/en-us/articles/207241115-How-to-Collecting-PhpStorm-WebStorm-Debug-Logs

@jakule Let's please make sure to include this info in our guide as well.

@jakule In my case, the agent is not running.....Could you instruct me how to enable it? Thank you anyway.

OS: windows 11, tsh version: 12.4.3, the output of "get-service ssh-agent" shows that the agent is stopped....The config generated by tsh is presented below:

Host *.main proxy.url UserKnownHostsFile C:\Users\name.tsh\known_hosts IdentityFile "C:\Users\name.tsh\keys\proxy.url\name" CertificateFile "C:\Users\name.tsh\keys\proxy.url\name-ssh\main-cert.pub" HostKeyAlgorithms [email protected],[email protected],[email protected]

Flags for all main hosts except the proxy

Host *.main !proxy.url Port 3022 ProxyCommand C:\Users\name\AppData\Local\Programs\teleport-connect\resources\bin\tsh.exe proxy ssh --cluster=main --proxy=proxy.url %r@%h:%p

We have similar issue with remote python interpreter in Pycharm.

Replication steps:

1. tsh login
2. tsh configure
3. verify that ssh works as proxy
4. open pycharm
5. create ssh connection and verify that is working
6. add remote python interpreter (the creation works and I'm able to select the virtual env)
7. run any test script (expect error)

Log from Pycharm:

2023-10-05 17:24:12,351 [5634444]   WARN - #c.i.s.i.s.sshj - <<global req for tcpip-forward>> woke to: net.schmizz.sshj.connection.ConnectionException: Global request [global req for tcpip-forward] failed

We have similar issue with remote python interpreter in Pycharm.

Replication steps:

1. tsh login
2. tsh configure
3. verify that ssh works as proxy
4. open pycharm
5. create ssh connection and verify that is working
6. add remote python interpreter (the creation works and I'm able to select the virtual env)
7. run any test script (expect error)

Log from Pycharm:

2023-10-05 17:24:12,351 [5634444]   WARN - #c.i.s.i.s.sshj - <<global req for tcpip-forward>> woke to: net.schmizz.sshj.connection.ConnectionException: Global request [global req for tcpip-forward] failed

i have the same error as yours , it is annoying,. cant use remote debug.

I can repro this on PyCharm 2024.1.1.

gus@apollo:~ % tctl status
Cluster       teleport.example.com
Version       15.3.0

gus@hades:~$ teleport version
Teleport v15.3.0 git:v15.3.0-0-g0a192a4 go1.21.9

Teleport logs:

May 10 12:14:54 hades teleport[4133068]: 2024-05-10T12:14:54-03:00 DEBU [SSH:NODE]  Received out-of-band request: &{Type:tcpip-forward WantReply:true Payload:[0 0 0 9 108 111 99 97 108 104 111 115 116 0 0 0 0] ch:<nil> mux:0xc000f9c7e0}. sshutils/server.go:621
May 10 12:14:54 hades teleport[4133068]: 2024-05-10T12:14:54-03:00 WARN             failed to handle tcpip forward request error:[
May 10 12:14:54 hades teleport[4133068]: ERROR REPORT:
May 10 12:14:54 hades teleport[4133068]: Original Error: *trace.AccessDeniedError unexpected user UID for the socket: 0
May 10 12:14:54 hades teleport[4133068]: Stack Trace:
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver_linux.go:56 github.com/gravitational/teleport/lib/srv/regular.validateListenerSocket
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1219 github.com/gravitational/teleport/lib/srv/regular.(*Server).listenTCPIP
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:2322 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleTCPIPForwardRequest
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1401 github.com/gravitational/teleport/lib/srv/regular.(*Server).HandleRequest
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/sshutils/server.go:638 github.com/gravitational/teleport/lib/sshutils.(*Server).HandleConnection.func2
May 10 12:14:54 hades teleport[4133068]:         runtime/asm_amd64.s:1650 runtime.goexit
May 10 12:14:54 hades teleport[4133068]: User Message: unexpected user UID for the socket: 0] regular/sshserver.go:1402
May 10 12:14:54 hades teleport[4133068]: 2024-05-10T12:14:54-03:00 INFO [AUDIT]     session.data addr.remote:192.168.64.203:49258 code:T2006I ei:2.147483646e+09 event:session.data login:gus namespace:default private_key_policy:none rx:1.4245882e+07 server_hostname:hades server_id:0065053d-e8e9-4880-aebb-e2d9499b681e sid: time:2024-05-10T15:14:54.325Z asset_tag:RG0HHJGJQP credential_id:eda45359-7377-40dc-9337-46076afc9ec3 device_id:dfa814c0-a5f4-4673-8b7a-3ded52538969 tx:4.629191e+07 uid:f91dd489-5588-4e3c-9c6b-5f4c0793860e user:webvictim user_kind:1 events/emitter.go:288
May 10 12:14:54 hades teleport[4133068]: 2024-05-10T12:14:54-03:00 DEBU [SUBSYSTEM] SFTP process finished regular/sftp.go:216
May 10 12:14:54 hades teleport[4133068]: 2024-05-10T12:14:54-03:00 DEBU             Subsystem &{0xc000444bd0 <nil> 0xc006a9eb00 0xc0013ad180 0xc002c0e120} finished with result: exit status 255. regular/sshserver.go:2068

PyCharm logs:

2024-05-10 12:14:53,265 [1716323]   INFO - #c.i.s.i.s.sshj - Will request `sftp` subsystem
2024-05-10 12:14:53,489 [1716547]   INFO - #c.i.s.i.s.sshj - Will request `sftp` subsystem
2024-05-10 12:14:53,688 [1716746]   INFO - #c.j.p.r.RemoteUpload - Uploading helpers from [/Applications/PyCharm.app/Contents/plugins/python/helpers, /Applications/PyCharm.app/Contents/plugins/python/helpers-pro] to /home/gus/.pycharm_helpers
2024-05-10 12:14:53,692 [1716750]   INFO - #c.i.s.i.s.sshj - Will request `sftp` subsystem
2024-05-10 12:14:53,883 [1716941]   INFO - #c.j.p.r.RemoteUpload - Remote helpers version is PY-241.15989.155, local helpers version is PY-241.15989.155
2024-05-10 12:14:53,945 [1717003]   INFO - #c.i.o.d.Logger - Plots on port: 63342
2024-05-10 12:14:53,945 [1717003]   INFO - #c.i.o.d.Logger - Interactive plots: 1
2024-05-10 12:14:53,952 [1717010]   INFO - #c.i.s.i.s.sshj - Will request `sftp` subsystem
2024-05-10 12:14:54,159 [1717217]   INFO - #c.j.p.r.RemoteUpload - Uploading helpers from [/Applications/PyCharm.app/Contents/plugins/python/helpers, /Applications/PyCharm.app/Contents/plugins/python/helpers-pro] to /home/gus/.pycharm_helpers
2024-05-10 12:14:54,163 [1717221]   INFO - #c.i.s.i.s.sshj - Will request `sftp` subsystem
2024-05-10 12:14:54,343 [1717401]   INFO - #c.j.p.r.RemoteUpload - Remote helpers version is PY-241.15989.155, local helpers version is PY-241.15989.155
2024-05-10 12:14:54,351 [1717409]   WARN - #c.i.s.i.s.sshj - <<global req for tcpip-forward>> woke to: net.schmizz.sshj.connection.ConnectionException: Global request [global req for tcpip-forward] failed
2024-05-10 12:14:54,352 [1717410]   WARN - #c.j.p.c.PydevConsoleRunnerImpl - Error running console
com.intellij.execution.ExecutionException: Failed to prepare environment.
	at com.jetbrains.plugins.remotesdk.target.ssh.target.SshRemoteEnvironmentRequest.prepareEnvironment(SshRemoteEnvironmentRequest.kt:109)
	at com.jetbrains.python.console.PydevConsoleRunnerImpl.createProcessUsingTargetsAPI(PydevConsoleRunnerImpl.java:579)
	at com.jetbrains.python.console.PydevConsoleRunnerImpl.initAndRun(PydevConsoleRunnerImpl.java:836)
	at com.jetbrains.python.console.PydevConsoleRunnerImpl$2.run(PydevConsoleRunnerImpl.java:327)
	at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:477)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:133)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$6(CoreProgressManager.java:528)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:250)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:221)
	at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:46)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:220)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:660)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:735)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:691)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:659)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:79)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:202)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$5(ProgressRunner.java:250)
	at com.intellij.openapi.progress.impl.ProgressRunner$ProgressRunnable.run(ProgressRunner.java:500)
	at com.intellij.util.concurrency.ChildContext$runAsCoroutine$1.invoke(propagation.kt:81)
	at com.intellij.util.concurrency.ChildContext$runAsCoroutine$1.invoke(propagation.kt:81)
	at com.intellij.util.concurrency.ChildContext.runAsCoroutine(propagation.kt:86)
	at com.intellij.util.concurrency.ChildContext.runAsCoroutine(propagation.kt:81)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$launchTask$18(ProgressRunner.java:466)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.intellij.ssh.SshTunnelCreationException: While creating remote tunnel for SshjSshConnection(gus@hades)@3f58de4d: :56942 <== localhost:<any free port>: Global request [global req for tcpip-forward] failed
	at com.intellij.ssh.impl.sshj.tunnels.RemoteForwarderCollection.wrapIntoTunnelCreationException(RemoteForwarderCollection.kt:71)
	at com.intellij.ssh.impl.sshj.tunnels.RemoteForwarderCollection.createWithRandomRemotePort(RemoteForwarderCollection.kt:57)
	at com.intellij.ssh.impl.sshj.SshjSshConnection.addRemoteTunnelWithRandomRemotePort(SshjSshConnection.kt:164)
	at com.jetbrains.plugins.remotesdk.target.ssh.target.SshEnvironment$Companion.create(SshEnvironment.kt:387)
	at com.jetbrains.plugins.remotesdk.target.ssh.target.SshRemoteEnvironmentRequest.prepareEnvironment(SshRemoteEnvironmentRequest.kt:97)
	... 32 more
Caused by: net.schmizz.sshj.connection.ConnectionException: Global request [global req for tcpip-forward] failed
	at net.schmizz.sshj.connection.ConnectionImpl.gotGlobalReqResponse(ConnectionImpl.java:220)
	at net.schmizz.sshj.connection.ConnectionImpl.handle(ConnectionImpl.java:140)
	at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:494)
	at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:113)
	at net.schmizz.sshj.transport.Decoder.received(Decoder.java:200)
	at net.schmizz.sshj.transport.Reader.run(Reader.java:60)
2024-05-10 12:14:54,364 [1717422]   WARN - #c.i.o.a.i.ActionToolbarImpl - 'PydevConsoleRunnerErrors' toolbar by default uses any focused component to update its actions. Toolbar actions that need local UI context would be incorrectly disabled. Please call toolbar.setTargetComponent() explicitly.
java.lang.Throwable: toolbar creation trace
	at com.intellij.openapi.actionSystem.impl.ActionToolbarImpl.<init>(ActionToolbarImpl.java:130)
	at com.intellij.openapi.actionSystem.impl.ActionManagerImplKt.createActionToolbarImpl(ActionManagerImpl.kt:1554)
	at com.intellij.openapi.actionSystem.impl.ActionManagerImplKt.access$createActionToolbarImpl(ActionManagerImpl.kt:1)
	at com.intellij.openapi.actionSystem.impl.ActionManagerImpl.createActionToolbar(ActionManagerImpl.kt:320)
	at com.intellij.openapi.actionSystem.impl.ActionManagerImpl.createActionToolbar(ActionManagerImpl.kt:304)
	at com.jetbrains.python.console.PydevConsoleRunnerImpl.showErrorsInConsole(PydevConsoleRunnerImpl.java:346)
	at com.jetbrains.python.console.PydevConsoleRunnerImpl$2.lambda$run$0(PydevConsoleRunnerImpl.java:335)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:308)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:792)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:733)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:761)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:699)
	at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$12(IdeEventQueue.kt:593)
	at com.intellij.openapi.application.impl.RwLockHolder.runWithoutImplicitRead(RwLockHolder.kt:105)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:593)
	at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:77)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:362)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:361)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:843)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:361)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:356)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:1022)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:106)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:1022)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(IdeEventQueue.kt:356)
	at com.intellij.openapi.application.impl.RwLockHolder.runIntendedWriteActionOnCurrentThread(RwLockHolder.kt:209)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:830)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:398)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

https://youtrack.jetbrains.com/issue/PY-43638/SSH-Remote-debugging-fails-while-creating-remote-tunnel

@webvictim it looks like you might be running into https://github.com/gravitational/teleport/issues/42254. The error trace and message that you posted is identical to what is reported there.

May 10 12:14:54 hades teleport[4133068]: Original Error: *trace.AccessDeniedError unexpected user UID for the socket: 0
May 10 12:14:54 hades teleport[4133068]: Stack Trace:
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver_linux.go:56 github.com/gravitational/teleport/lib/srv/regular.validateListenerSocket
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1219 github.com/gravitational/teleport/lib/srv/regular.(*Server).listenTCPIP
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:2322 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleTCPIPForwardRequest
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1401 github.com/gravitational/teleport/lib/srv/regular.(*Server).HandleRequest
May 10 12:14:54 hades teleport[4133068]:         github.com/gravitational/teleport/lib/sshutils/server.go:638 github.com/gravitational/teleport/lib/sshutils.(*Server).HandleConnection.func2
May 10 12:14:54 hades teleport[4133068]:         runtime/asm_amd64.s:1650 runtime.goexit
May 10 12:14:54 hades teleport[4133068]: User Message: unexpected user UID for the socket: 0] regular/sshserver.go:1402

Yes, it looks like that same code handles all tcpip-forward events. Maybe it'll also be fixed by https://github.com/gravitational/teleport/pull/43069?

Yes, it looks like that same code handles all tcpip-forward events. Maybe it'll also be fixed by #43069?

I've reproduced this issue as well, I'll test the fix now.