kex_exchange_identification: Connection closed by remote host

[richardleeps]

trafficstars

Describe the bug Unable to connect to ssh target through an egress worker. Getting the error message "Connection closed by 127.0.0.1 port 52471" The SSH target is a managed application. We are able to connect to other ssh targets such as ubuntu VM's through the same egress worker.

Error message when connection with the client

ssh 127.0.0.1 -p 52471 -o NoHostAuthenticationForLocalhost=yes

debug1: Local version string SSH-2.0-OpenSSH_9.6
kex_exchange_identification: Connection closed by remote host
Connection closed by 127.0.0.1 port 52471

When we connect from the egress worker directly to the target its working

ssh [email protected]

debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.11 pat OpenSSH* compat 0x04000000

To Reproduce Steps to reproduce the behavior:

1. Start Boundary Client
2. Connect the target
3. Error is there MacBook-Pro-2 ~ % ssh 127.0.0.1 -p 52471 -o NoHostAuthenticationForLocalhost=yes Connection closed by 127.0.0.1 port 52471

Expected behavior A succesfull connection directly from a boundary client

Additional context Similar issue was dropped (https://github.com/hashicorp/boundary/issues/765) but was closed due to no response

[AdamBouhmad]

Hey @richardleeps, what version of Boundary are you running(client & controller)? Also curious if you're able to reproduce on 0.16.

[richardleeps]

@AdamBouhmad - All boundary nodes are on v0.16.0+ent I just tried and still the same result.

If I compare the SSH debug this is what I see

ssh 127.0.0.1 -p 63924 -o NoHostAuthenticationForLocalhost=yes -vv
OpenSSH_9.6p1, LibreSSL 3.3.6
<snip>
debug1: Local version string SSH-2.0-OpenSSH_9.6
kex_exchange_identification: Connection closed by remote host
Connection closed by 127.0.0.1 port 63924

While direct from the egress worker this is wat we get

ssh [email protected] -vvv
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
<snip>
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.11 pat OpenSSH* compat 0x04000000

[AdamBouhmad]

Thanks for the quick response @richardleeps. Do you mind sharing over your worker logs so we can dig into this further? Feel free to email me at [email protected].

[TommyStarK]

I'm having the exact same issue. I'm trying to inject ssh credentials into sessions to connect to a vm. The virtual machine has been configured to accept ssh connections if the key is signed by a specific CA. Boundary is configured to talk to a Vault to issue these signed ssh keypairs.

I'm running the following version on the client and server side with a valid license for the server:

Version information:
  Build Date:          2024-04-29T15:21:18Z
  Git Revision:        ad952555caf942873e0b410eecc69feda81fdb64
  Metadata:            ent
  Version Number:      0.16.0+ent

The bootstrap and configuration of the service has been tested multiple times and worked well. However, I'm experiencing connections issues and the only log I got is:

• on the worker side (log level = debug)

[INFO]  system event: version=v0.1 op="command_routing.(CommandServiceServer).SendCommand" data:msg="Invoking command COMMAND_UNSPECIFIED for worker: w_HpwMEmE08C"

• on the client side

kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 34483