allow_agent and loading same keys from files

[cvicente]

Greetings.

Thank you very much for sharing this useful module.

While testing with "allow_agent=True", I found that sshtunnel successfully gets the private keys from the agent, but then logs an error when it cannot decrypt the same keys from files:

2021-01-12 11:05:10,415| ERROR   | Password is required for key /Users/username/.ssh/id_rsa

I feel that it should be "either or". That is, if allow_agent=True, then it should not try to load keys from the ssh config directory. Thoughts?

[pahaz]

Could you please provide the whole logs with sshtunnel.DEFAULT_LOGLEVEL = 1? It looks like a bug. Similar to #226

[cvicente]

Thanks.

Here's the output:

$ sshtunnel -U my_user -R 10.1.174.11:443 -L localhost:10443 -vvv bastion.example.net
2021-02-04 10:13:06,037| INF | MainThrea/1060@sshtunnel | 2 keys loaded from agent
2021-02-04 10:13:06,038| ERR | MainThrea/1315@sshtunnel | Password is required for key /Users/my_user/.ssh/id_rsa
2021-02-04 10:13:06,039| ERR | MainThrea/1315@sshtunnel | Password is required for key /Users/my_user/.ssh/id_dsa
2021-02-04 10:13:06,039| INF | MainThrea/1117@sshtunnel | 2 key(s) loaded
2021-02-04 10:13:06,040| INF | MainThrea/0981@sshtunnel | Connecting to gateway: bastion.example.net:22 as user 'my_user'
2021-02-04 10:13:06,040| DEB | MainThrea/0984@sshtunnel | Concurrent connections allowed: False
2021-02-04 10:13:06,040| DEB | MainThrea/1401@sshtunnel | Trying to log in with key: b'xxxxxx'
2021-02-04 10:13:06,457| DEB | MainThrea/1205@sshtunnel | Transport socket info: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 0), timeout=0.1
2021-02-04 10:13:07,799| INF | Srv-10443/1435@sshtunnel | Opening tunnel: 127.0.0.1:10443 <> 10.1.174.11:443


            Press <Ctrl-C> or <Enter> to stop!

[bdoyle0182]

After digging through the code a couple work arounds / bugs I've figured out for this.

1. If you don't set ssh_pkey, it still gets set if there is an IdentityFile defined in your ssh config and that is loaded and then is attempted to load from a file.
2. If you don't set host_pkey_directories, it will still add the default ssh directory ~.ssh to search and find id_* files and try to load them. You can workaround this by setting this to host_pkey_directories=[] in the creation of SSHTunnelForwarder.
3. In replace of 1. since I don't want to force my clients of my service to have to comment out their IdentityFile, you can import paramiko and do ssh_pkey=paramiko.agent.Agent().get_keys() which is exactly what's done if allow_agent=True anyhow to load the keys. So technically it will be loaded twice but still much better than forcing clients to edit their config file.

So to fix the problem you'll need to do 2. and (1. or 3.).

But I'll emphasize the reason you opened the issue. Certain options being set should not impact other options being set so this needs to be fixed. If allow_agent=True all of these other options that get set once in code through system defaults with no knowledge to the user should not be happening.

Also if you don't set either ssh_pkey or host_pkey_directories and you have a password protected id_* file in ~/.ssh, then when it searches that directory even though you didn't tell it to load anything through host_pkey_directories, it's going to throw an uncaught exception causing ssh_loaded_pkeys to go back to none even though they had already been properly loaded through the agent. This is even worse because the connection fails in this case with raise ValueError('No password or public key available!') rather than just emit an improper Password is required for key log. I believe this is what's happening with the other issues that were linked.

Hope this helps with fixing the issue @pahaz