Hostname used withing Host entries in ssh config

[ghost]

On Debian 10 "buster", amd64, using rex from backports, version "(R)?ex 1.12.2". Those points where discussed on IRC.

Describe the bug

Rex uses the Hostname sub-key of a Host section in ~/.ssh/config

How to reproduce it

Steps to reproduce the behavior:

1. Create 1 proxy machine "proxy" with 2 NICs and a running ssh-server instance; 1.1 connect 1 NIC so that it can be reached from LAN, and give it, for example, IP 192.168.1.200/16 1.2 connect the other one so that it can not be reached from LAN and give it IP 10.0.0.1/16 1.3 create a user "bouncer", with no password and nologin shell; 1.4 give "bouncer" a file ~bouncer/.ssh/authorized_keys filled with a public key "vmproxy";
2. Create 1 target machine "target" which only NIC is only reachable by "proxy"; 2.1 create a normal user "user" 2.2 give "user" a file ~user/.ssh/authorized_keys filled with a public key "vmlan"; 2.3 give NIC the IP 10.0.0.10/16
3. on host, create a ~/.ssh/config file such as:

Host proxy
  Hostname 192.168.1.200
  User bouncer
  IdentityFile ~/.ssh/vmproxy

Host target
  Hostname 10.0.0.10
  User user
  IdentityFile ~/.ssh/vmlan
  ProxyJump proxy

4. connect to proxy via ssh. It should work.
5. connect to target via ssh. It should also work.
6. run rex -H target -u u user -e say run uptime <= will fail
7. on host, modify a ~/.ssh/config file such as:

Host proxy
  Hostname 192.168.1.200
  User bouncer
  IdentityFile ~/.ssh/vmproxy

Host target
  Hostname 10.0.0.10
Host 10.0.0.10
  User user
  IdentityFile ~/.ssh/vmlan
  ProxyJump proxy

8. run rex -H target -u u user -e say run uptime <= will work

Expected behavior

step 6 should behave the same as step 8

Circumstances

• Rex version: (R)?ex 1.12.2
• Perl version: This is perl 5, version 28, subversion 1 (v5.28.1) built for x86_64-linux-gnu-thread-multi (with 65 registered patches, see perl -V for more detail)
• OS running rex: Debian 10 buster, amd64
• OS managed by rex: same.
• How rex was installed: apt-get

Debug log

[31m[2020-12-13 21:23:18] DEBUG - This is Rex version: 1.12.2
[0m[31m[2020-12-13 21:23:18] DEBUG - Command Line Parameters
[0m[31m[2020-12-13 21:23:18] DEBUG - 	e = say
[0m[31m[2020-12-13 21:23:18] DEBUG - 	u = user
[0m[31m[2020-12-13 21:23:18] DEBUG - 	H = sfiles
[0m[31m[2020-12-13 21:23:18] DEBUG - 	d = 1
[0m[31m[2020-12-13 21:23:18] DEBUG - Creating lock-file (Rexfile.lock)
[0m[31m[2020-12-13 21:23:18] DEBUG - Loading Rexfile
[0m[31m[2020-12-13 21:23:18] DEBUG - Creating new distribution class of type: Base
[0m[31m[2020-12-13 21:23:18] DEBUG - new distribution class of type Rex::TaskList::Base created.
[0m[31m[2020-12-13 21:23:18] DEBUG - Initializing Logger from parameters found in Rexfile
[0m[31m[2020-12-13 21:23:18] DEBUG - Executing command line code
[0m[31m[2020-12-13 21:23:18] DEBUG - 	say
[0m[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
[0m[31m[2020-12-13 21:23:18] DEBUG - Creating task: eval-line
[0m[31m[2020-12-13 21:23:18] DEBUG - 	server: sfiles
[0m[31m[2020-12-13 21:23:18] DEBUG - Found Net::OpenSSH and Net::SFTP::Foreign - using it as default
[0m[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
[0m[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
[0m[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
[0m[32m[2020-12-13 21:23:18] INFO - Running task eval-line on sfiles
[0m[31m[2020-12-13 21:23:18] DEBUG - Checking for a private key in .ssh/config
[0m[31m[2020-12-13 21:23:18] DEBUG - Rex::Group::Entry::Server (private_key): returning ~/.ssh/vmlan
[0m[31m[2020-12-13 21:23:18] DEBUG - Checking for a public key in .ssh/config
[0m[31m[2020-12-13 21:23:18] DEBUG - Rex::Group::Entry::Server (public_key): returning ~/.ssh/vmlan.pub
[0m[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = '';

[0m[31m[2020-12-13 21:23:18] DEBUG - Auth-Information inside Task:
[0m[31m[2020-12-13 21:23:18] DEBUG - public_key => [[~/.ssh/vmlan.pub]]
[0m[31m[2020-12-13 21:23:18] DEBUG - port => [[]]
[0m[31m[2020-12-13 21:23:18] DEBUG - sudo_password => [[**********]]
[0m[31m[2020-12-13 21:23:18] DEBUG - password => [[%s]]
[0m[31m[2020-12-13 21:23:18] DEBUG - auth_type => [[try]]
[0m[31m[2020-12-13 21:23:18] DEBUG - user => [[user]]
[0m[31m[2020-12-13 21:23:18] DEBUG - sudo => [[]]
[0m[31m[2020-12-13 21:23:18] DEBUG - private_key => [[~/.ssh/vmlan]]
[0m[31m[2020-12-13 21:23:18] DEBUG - Using Net::OpenSSH for connection
[0m[31m[2020-12-13 21:23:18] DEBUG - Using user: user
[0m[31m[2020-12-13 21:23:18] DEBUG - Connecting to 10.0.1.10:22 (user)
[0m[31m[2020-12-13 21:23:18] DEBUG - get_openssh_opt()
[0m[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = {};

[0m[31m[2020-12-13 21:23:18] DEBUG - OpenSSH: key_auth or not defined: 10.0.1.10:22 - user
[0m[31m[2020-12-13 21:23:18] DEBUG - OpenSSH options: 
[0m[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = [
          '10.0.1.10',
          'user',
          'user',
          'port',
          22,
          'master_opts',
          [
            '-o',
            'ConnectTimeout=2',
            '-o',
            'LogLevel=QUIET'
          ],
          'default_ssh_opts',
          $VAR1->[6]
        ];

[0m[31m[2020-12-13 21:23:18] DEBUG - OpenSSH constructor options: 
[0m[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = {};

[0m[31m[2020-12-13 21:23:18] DEBUG - Trying following auth types:
[0m[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = [
          'key',
          'pass'
        ];

[0m[33m[2020-12-13 21:23:31] WARN - Can't authenticate against 10.0.1.10 (unable to establish master SSH connection: master process exited unexpectedly)
[0m[31m[2020-12-13 21:23:31] DEBUG - Connections in queue: 1
[0m[31m[2020-12-13 21:23:31] DEBUG - Destroying all cached os information
[0m[31m[2020-12-13 21:23:31] DEBUG - Waiting for children to finish
[0m[31m[2020-12-13 21:23:31] DEBUG - Need to reinitialize connections.
[0m[31m[2020-12-13 21:23:31] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
[0m[31m[2020-12-13 21:23:31] ERROR - 1 out of 1 task(s) failed:
[0m[31m[2020-12-13 21:23:31] ERROR - 	eval-line failed on sfiles
[0m[31m[2020-12-13 21:23:31] ERROR - 		Couldn't authenticate against sfiles. It may be caused by one or more of:
[0m[31m[2020-12-13 21:23:31] ERROR - 		 - wrong username, password, key or passphrase
[0m[31m[2020-12-13 21:23:31] ERROR - 		 - changed remote host key
[0m[31m[2020-12-13 21:23:31] ERROR - 		 at /usr/share/perl5/Rex/TaskList/Base.pm line 342.
[0m[31m[2020-12-13 21:23:31] DEBUG - Removing lockfile
[0m[31m[2020-12-13 21:23:31] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base

[ferki]

Thanks for the patience, and for the highly detailed report! :+1: Providing a workaround example for others is also highly appreciated :heart:

I couldn't do a fully detailed examination of the problem yet, but I still wanted to make some notes for whoever looking at this issue next (probably me ^^):

• I assume that the IP address mismatch between the sample ssh config (10.0.0.10) and the provided debug log (10.0.1.10) is only due to a typo.
• I believe that the behavior might be slightly different depending on the SSH backend module in use (Net::SSH2 vs Net::OpenSSH). The latter uses the OpenSSH binary under the hood, so that should explain why it can pick up the correct bits from ssh config when using the workaround (which I expect to still fail with Net::SSH2 - FIXME).
• For the same reason, I believe the correct fix would be to skip ssh config parsing when Net::OpenSSH backend is used (as proposed on #1229), but it might also be possible to improve the current approach before that bigger change.