`ProxyCommand` to proxy-jump into the server is not working

[allencho1222]

What Operating System(s) are you seeing this problem on?

macOS

Which Wayland compositor or X11 Window manager(s) are you using?

No response

WezTerm version

20230929-093024-5e0fd1a0

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

No, and I'll explain why below

Describe the bug

I have tried to access the server (target_server) behind the proxy server (proxy_server).

My ssh config (~/.ssh/config) file is:

Host target_server
  HostName 127.0.0.1
  User root
  Port 20023
  #ServerAliveInterval 30
  #ProxyJump proxy_server
  ProxyCommand ssh -W %h:%p proxy_server

Host proxy_server
  HostName xxx.xxx.xxx.xxx
  User imuser
  #ServerAliveInterval 60
  # jupyterlab docker container
  #LocalForward 10088 127.0.0.1:10088
  #LocalForward 20024 127.0.0.1:20024
  #LocalForward 15213 127.0.0.1:15213
  #LocalForward 15214 127.0.0.1:15213
  #LocalForward 18888 127.0.0.1:18888

I have tried following commands that successfully access servers:

1. ssh proxy_server
2. ssh target_server
3. ssh -p 20023 -o "ProxyCommand=ssh -W %h:%p proxy_server" [email protected]
4. wezterm ssh proxy_server

However, wezterm ssh fails to access target_server:

1. wezterm ssh target_server
2. wezterm ssh -oProxyCommand="ssh -W %h:%p proxy_server" [email protected]:20023

To Reproduce

No response

Configuration

Nothing special, but just in case:

-- Pull in the wezterm API
local wezterm = require 'wezterm'

-- This table will hold the configuration.
local config = {}

-- In newer versions of wezterm, use the config_builder which will
-- help provide clearer error messages
if wezterm.config_builder then
  config = wezterm.config_builder()
end

-- This is where you actually apply your config choices

-- For example, changing the color scheme:
config.color_scheme = 'AdventureTime'

config.ssh_domains = {
  {
    -- This name identifies the domain
    name = 'dev.linux',
    -- The hostname or address to connect to. Will be used to match settings
    -- from your ssh config file
    remote_address = 'dev-linux',
    -- The username to use on the remote host
    -- username = 'wez',
  },
}

-- and finally, return the configuration to wezterm
return config

Expected Behavior

wezterm ssh target_server must work.

Logs

Here's the log from wezterm ssh -v target_server (It seems that "proxycommand" is not what it has to be):

sungjun@sungjuns-MacBook-Pro ~ % wezterm ssh -v target_server
00:06:59.831  INFO   mux::ssh > Using ssh config: {
    "forwardagent": "yes",
    "hostname": "127.0.0.1",
    "identityagent": "/Users/sungjun/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock",
    "identityfile": "/Users/sungjun/.ssh/id_dsa /Users/sungjun/.ssh/id_ecdsa /Users/sungjun/.ssh/id_ed25519 /Users/sungjun/.ssh/id_rsa",
    "port": "20023",
    "proxycommand": "ssh -W target_server:20023 proxy_server",
    "sendenv": "LANG LC_*",
    "user": "root",
    "userknownhostsfile": "/Users/sungjun/.ssh/known_hosts /Users/sungjun/.ssh/known_hosts2",
    "wezterm_ssh_backend": "libssh",
    "wezterm_ssh_verbose": "true",
}
00:06:59.831  INFO   libssh::ssh_config_parse_file > Reading configuration data from /Users/sungjun/.ssh/config
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ForwardAgent, line: 3
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 9
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 15
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 21
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 28
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 35
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 37
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 43
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 46
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 48
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 50
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 56
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 62
00:06:59.831  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 86
00:06:59.832  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 92
00:06:59.832  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 93
00:06:59.832  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 100
00:06:59.832  INFO   libssh::ssh_config_parse_file > Reading configuration data from /etc/ssh/ssh_config
00:06:59.832  INFO   libssh::local_parse_file      > Cannot find file /etc/ssh/ssh_config.d/* to load
00:06:59.832  INFO   libssh::ssh_config_parse_line > Unapplicable option: SendEnv, line: 55
00:06:59.832  INFO   libssh::ssh_connect           > libssh 0.8.90 (c) 2003-2023 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
00:06:59.832  INFO   libssh::ssh_connect           > Socket connecting, now waiting for the callbacks to work
00:06:59.832  INFO   libssh::ssh_connect           > Actual timeout : 10000
00:06:59.832  INFO   libssh::ssh_socket_pollcallback > Received POLLOUT in connecting state
00:06:59.832  INFO   libssh::socket_callback_connected > Socket connection callback: 1 (0)
00:06:59.832  INFO   libssh::ssh_client_connection_callback > session_state=2
00:06:59.832  INFO   libssh::ssh_socket_unbuffered_write    > Enabling POLLOUT for socket
channel 0: open failed: connect failed: Temporary failure in name resolution
stdio forwarding failed
00:07:00.194  INFO   libssh::ssh_socket_exception_callback  > Socket exception callback: 1 (0)
00:07:00.194  INFO   libssh::ssh_socket_exception_callback  > Socket error: disconnected
00:07:00.194  INFO   libssh::ssh_client_connection_callback > session_state=9
00:07:00.194  INFO   libssh::ssh_connect                    > current state : 9

Here's the output from new terminal window popped after wezterm ssh -v target_server:

Using libssh-rs to connect to [email protected]:20023
Error: Connecting to 127.0.0.1:20023: Fatal: Socket error: disconnected

⚠️  Process RemoteSshDomain didn't exit cleanly
Exited with code 1.
This message is shown because exit_behavior="CloseOnCleanExit"

Because "proxycommand" seems to be wrong, I also show another log from wezterm ssh -v -oProxyCommand="ssh -W %h:%p proxy_server" [email protected]:20023 which also fails to access [email protected]:20023.

sungjun@sungjuns-MacBook-Pro ~ % wezterm ssh -v -oProxyCommand="ssh -W %h:%p proxy_server" [email protected]:20023
00:17:24.602  INFO   mux::ssh > Using ssh config: {
    "forwardagent": "yes",
    "hostname": "127.0.0.1",
    "identityagent": "/Users/sungjun/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock",
    "identityfile": "/Users/sungjun/.ssh/id_dsa /Users/sungjun/.ssh/id_ecdsa /Users/sungjun/.ssh/id_ed25519 /Users/sungjun/.ssh/id_rsa",
    "port": "20023",
    "proxycommand": "ssh -W %h:%p proxy_server",
    "sendenv": "LANG LC_*",
    "user": "root",
    "userknownhostsfile": "/Users/sungjun/.ssh/known_hosts /Users/sungjun/.ssh/known_hosts2",
    "wezterm_ssh_backend": "libssh",
    "wezterm_ssh_verbose": "true",
}
00:17:24.603  INFO   libssh::ssh_config_parse_file > Reading configuration data from /Users/sungjun/.ssh/config
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ForwardAgent, line: 3
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 9
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 15
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 21
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 28
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 35
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 37
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 43
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 46
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 48
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 50
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 56
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 62
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 86
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unsupported option: ServerAliveInterval, line: 92
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 93
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: LocalForward, line: 100
00:17:24.603  INFO   libssh::ssh_config_parse_file > Reading configuration data from /etc/ssh/ssh_config
00:17:24.603  INFO   libssh::local_parse_file      > Cannot find file /etc/ssh/ssh_config.d/* to load
00:17:24.603  INFO   libssh::ssh_config_parse_line > Unapplicable option: SendEnv, line: 55
00:17:24.603  INFO   libssh::ssh_connect           > libssh 0.8.90 (c) 2003-2023 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
00:17:24.603  INFO   libssh::ssh_connect           > Socket connecting, now waiting for the callbacks to work
00:17:24.603  INFO   libssh::ssh_connect           > Actual timeout : 10000
00:17:24.604  INFO   libssh::ssh_socket_pollcallback > Received POLLOUT in connecting state
00:17:24.604  INFO   libssh::socket_callback_connected > Socket connection callback: 1 (0)
00:17:24.604  INFO   libssh::ssh_client_connection_callback > session_state=2
00:17:24.604  INFO   libssh::ssh_socket_unbuffered_write    > Enabling POLLOUT for socket
Bad stdio forwarding specification '%h:%p'
00:17:24.612  INFO   libssh::ssh_socket_exception_callback  > Socket exception callback: 1 (0)
00:17:24.612  INFO   libssh::ssh_socket_exception_callback  > Socket error: disconnected
00:17:24.612  INFO   libssh::ssh_client_connection_callback > session_state=9
00:17:24.612  INFO   libssh::ssh_connect                    > current state : 9

Anything else?

No response

[aur3l14no]

Hi, I just ran into the same problem. Luckily there is a workaround.

Setup the host bar in your ssh config then add the following to wezterm config.

config.unix_domains = {
	{
		name = "foo",
		proxy_command = { "ssh", "-T", "bar", "wezterm", "cli", "proxy" },
	},
}

Then you can connect using wezterm connect foo.

I got this solution from #1568 and #1578. You might want to check them out as well.

[allencho1222]

@aur3l14no Thank you for your response. I will check them out.

[Davidyz]

It appears that the solution proposed by @aur3l14no requires the same version of wezterm to be installed on both the client and the server. This is not possible for my use case (the servers are lab machines in my university network and can only be accessed via ssh proxyjump if the client is not connected to the university LAN. Also, I do not have sudo access to those machines so I cannot install wezterm on the remote machines). Therefore I'd still like to kindly request support for ProxyJump.

[cxwx]

same issue with proxycommand

[SuperSandro2000]

I receive another strange error:

> wezterm connect krypton
mux_client_request_session: session request failed: Session open refused by peer
ControlSocket /home/user/.ssh/sockets/abc12346 already exists, disabling multiplexing

I have a ssh config on the remote host that enabled ControlMaster but why does wezterm want to ssh again?