Does not use unlocked key from ssh-agent? "Passphrase must be specified to import encrypted private keys"

[yarikoptic]

I am new to asyncssh. Arrived to it through pynwb -> fsspec -> sshfs path ;-) I am a heavy used of ssh-agent to keep my keys loaded (with timeout) and according to description asyncssh seems to support interacting with the agent. But it seems to not quite work:

NB borrowed unsync oneliner snippet from another issue, not my achievement

Getting that asyncssh.public_key.KeyImportError (click to expand)

(git)lena:~/proj/misc/asyncssh[develop]
$> python3 -c "from unsync import unsync;import asyncssh;print(asyncssh.__name__, asyncssh.__version__);exec('@unsync\nasync def f(func,*args,**opts):\n  return await func(*args,**opts)\n');print(f(asyncssh.connect, '127.0.0.1', login_timeout=10).result())"
asyncssh 2.11.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yoh/deb/gits/pkg-exppsy/pynwb-upstream/venvs/dev3/lib/python3.10/site-packages/unsync/unsync.py", line 144, in result
    return self.concurrent_future.result(*args, **kwargs)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "<string>", line 3, in f
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7718, in connect
    new_options = cast(SSHClientConnectionOptions, await _run_in_executor(
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 519, in _run_in_executor
    return await loop.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 6212, in __init__
    super().__init__(options=options, last_config=last_config, **kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/misc.py", line 350, in __init__
    self.prepare(**self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7066, in prepare
    load_keypairs(cast(KeyPairListArg, client_keys), passphrase,
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3458, in load_keypairs
    read_private_key_and_certs(key_to_load, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3272, in read_private_key_and_certs
    key, cert = import_private_key_and_certs(read_file(filename), passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3145, in import_private_key_and_certs
    key, end = _decode_private(data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2750, in _decode_private
    key = _decode_pem_private(pem_name, headers, data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2665, in _decode_pem_private
    raise KeyImportError('Passphrase must be specified to import '
asyncssh.public_key.KeyImportError: Passphrase must be specified to import encrypted private keys

although can login just fine with regular ssh:

$> ssh 127.0.0.1 echo done
done

and ssh-add -l shows a number of keys loaded/available. version of asyncssh v2.11.0-32-g7ec0e74 straight from current develop.

Thanks in advance for guidance on how to use asyncssh "more properly" or what debugging information to provide to make it turnkey for me.

[ronf]

The error suggests that you have an encrypted version of a private key that AsyncSSH is trying to load, meaning it is could be one of the default keys in your ".ssh" directory. However, the call to load_keypairs at line 3458 in asyncssh/public_key.py takes an ignore_encrypted argument which is supposed to skip over any encrypted private keys when you don't specify a passphrase when opening a new SSH connection, and this should happen automatically when loading default keys.

Do you have an SSH config file which might have references to private key files which are encrypted? If so, you may need to specify ignore_encrypted=True explicitly to tell AsyncSSH that it's ok to skip over those key files specified in your SSH config. Alternately, you could take those entries out of .ssh/config, and things should still work if those keys are already loaded into your SSH agent.

[yarikoptic]

well done @ronf ! It does relate to ~.ssh/config since

moving it aside "resolves" the issue

$> ssh -p 22 127.0.0.1 echo did it
did it

$> python3 -c "from unsync import unsync;import getpass,asyncssh;print(asyncssh.__name__, asyncssh.__version__);exec('@unsync\nasync def f(func,*args,**opts):\n  return await func(*args,**opts)\n');print(f(asyncssh.connect, '127.0.0.1', port=22, login_timeout=10).result())"
asyncssh 2.11.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yoh/proj/misc/asyncssh/venvs/dev3/lib/python3.10/site-packages/unsync/unsync.py", line 144, in result
    return self.concurrent_future.result(*args, **kwargs)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "<string>", line 3, in f
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7718, in connect
    new_options = cast(SSHClientConnectionOptions, await _run_in_executor(
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 519, in _run_in_executor
    return await loop.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 6212, in __init__
    super().__init__(options=options, last_config=last_config, **kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/misc.py", line 350, in __init__
    self.prepare(**self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7066, in prepare
    load_keypairs(cast(KeyPairListArg, client_keys), passphrase,
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3458, in load_keypairs
    read_private_key_and_certs(key_to_load, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3272, in read_private_key_and_certs
    key, cert = import_private_key_and_certs(read_file(filename), passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3145, in import_private_key_and_certs
    key, end = _decode_private(data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2750, in _decode_private
    key = _decode_pem_private(pem_name, headers, data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2665, in _decode_pem_private
    raise KeyImportError('Passphrase must be specified to import '
asyncssh.public_key.KeyImportError: Passphrase must be specified to import encrypted private keys
(dev3) 1 42302 ->1.....................................:Fri 05 Aug 2022 
$> mv ~/.ssh/config{,.aside}

$> ssh -p 22 127.0.0.1 echo did it
did it

(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> python3 -c "from unsync import unsync;import getpass,asyncssh;print(asyncssh.__name__, asyncssh.__version__);exec('@unsync\nasync def f(func,*args,**opts):\n  return await func(*args,**opts)\n');print(f(asyncssh.connect, '127.0.0.1', port=22, login_timeout=10).result())"
asyncssh 2.11.0
<asyncssh.connection.SSHClientConnection object at 0x7f8065c19c60>

$> mv ~/.ssh/config{.aside,}

adding `ignore_encrypted=True` also resolves it

$> python3 -c "from unsync import unsync;import getpass,asyncssh;print(asyncssh.__name__, asyncssh.__version__);exec('@unsync\nasync def f(func,*args,**opts):\n  return await func(*args,**opts)\n');print(f(asyncssh.connect, '127.0.0.1', port=22, login_timeout=10, ignore_encrypted=True).result())"
asyncssh 2.11.0
<asyncssh.connection.SSHClientConnection object at 0x7fd96d5c9cc0>

and yes -- I had reference to other private keys... but it is really the referencing of the key which is loaded into ssh-agent which already causes the error:

$> mv ~/.ssh/config{,.aside}; echo -e 'Host *\n IdentityFile %d/.ssh/id_rsa' > ~/.ssh/config

$> cat ~/.ssh/config
Host *
 IdentityFile %d/.ssh/id_rsa

$> ssh-add -l | grep 'ssh/id_rsa '
2048 SHA256:bFbba3dSW4DnHrEDITEDJUSTINCASE /home/yoh/.ssh/id_rsa (RSA)

(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> python3 -c "from unsync import unsync;import getpass,asyncssh;print(asyncssh.__name__, asyncssh.__version__);exec('@unsync\nasync def f(func,*args,**opts):\n  return await func(*args,**opts)\n');print(f(asyncssh.connect, '127.0.0.1', port=22, login_timeout=10).result())"
asyncssh 2.11.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yoh/proj/misc/asyncssh/venvs/dev3/lib/python3.10/site-packages/unsync/unsync.py", line 144, in result
    return self.concurrent_future.result(*args, **kwargs)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "<string>", line 3, in f
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7718, in connect
    new_options = cast(SSHClientConnectionOptions, await _run_in_executor(
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 519, in _run_in_executor
    return await loop.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 6212, in __init__
    super().__init__(options=options, last_config=last_config, **kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/misc.py", line 350, in __init__
    self.prepare(**self.kwargs)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/connection.py", line 7066, in prepare
    load_keypairs(cast(KeyPairListArg, client_keys), passphrase,
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3458, in load_keypairs
    read_private_key_and_certs(key_to_load, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3272, in read_private_key_and_certs
    key, cert = import_private_key_and_certs(read_file(filename), passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 3145, in import_private_key_and_certs
    key, end = _decode_private(data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2750, in _decode_private
    key = _decode_pem_private(pem_name, headers, data, passphrase)
  File "/home/yoh/proj/misc/asyncssh/asyncssh/public_key.py", line 2665, in _decode_pem_private
    raise KeyImportError('Passphrase must be specified to import '
asyncssh.public_key.KeyImportError: Passphrase must be specified to import encrypted private keys

I guess ideally asyncssh should avoid/skip trying to load those keys there if already known to ssh-agent.

[ronf]

Is that how OpenSSH currently works, skipping loading of encrypted private keys referenced in the config if they're already in the agent, or does it just always ignore such keys when a passphrase is not specified regardless of what's in the agent?

To only ignore keys loaded into the agent would probably require the '.pub' file for those keys to be present, at least for older key formats (before OpenSSH added its own private key format). Without either that or a '-cert.pub' file to go with the private key, there'd be no way to know if the key was already in the agent or not, whereas normally you don't actually need the '.pub' file for a private key to be usable. So, there'd still be cases where adding a reference to an encrypted private key in the config could trigger this error even when that key was already in the agent.

If encrypted keys in the config are always ignored by OpenSSH when no passphrase is specified, it would be pretty easy to change AsyncSSH to have that same behavior. It already does this for the default key names. I just didn't want to do it for something which was explicitly configured. However, I could perhaps do it for default keys and keys in the default config file (or perhaps any config file) but not for keys that you pass in explicitly as an argument, if that better matches OpenSSH.

[yarikoptic]

I think the best would be to look at openssh code and/or ask openssh developers (or may be it is RTFM).

Meanwhile: having .pub does not make it not use that key AFAIK, but does change the order in which it gets offered. It seems that without .pub it gets offered 2nd (after I moved it aside). I think it relies on key signature to match it against the one already in the ssh-agent

(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> ssh -vvv -p 22 127.0.0.1 echo did it 2>&1 | grep id
debug1: identity file /home/yoh/.ssh/id_rsa type 0
debug1: identity file /home/yoh/.ssh/id_rsa-cert type -1
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 5 keys
debug1: Will attempt key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug1: Will attempt key: /home/yoh/.ssh/id_rsa_mailcheck RSA SHA256:I8wESENSORED agent
debug1: Offering public key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug1: Server accepts key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug2: client_session2_setup: id 0
debug1: Sending command: echo did it
debug2: channel_input_status_confirm: type 99 id 0
did it
(dev3) 1 42353.....................................:Fri 05 Aug 2022 10:01:05 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> grep cert ~/.ssh/config
(dev3) 1 42354 ->1.....................................:Fri 05 Aug 2022 10:01:43 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> ls -ld /home/yoh/.ssh/id_rsa-cert
ls: cannot access '/home/yoh/.ssh/id_rsa-cert': No such file or directory
(dev3) 1 42355 ->2.....................................:Fri 05 Aug 2022 10:01:49 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> strace -f -o /tmp/ssh-strace ssh -vvv -p 22 127.0.0.1 echo did it 2>&1 | grep id
debug1: identity file /home/yoh/.ssh/id_rsa type 0
debug1: identity file /home/yoh/.ssh/id_rsa-cert type -1
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 5 keys
debug1: Will attempt key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug1: Will attempt key: /home/yoh/.ssh/id_rsa_mailcheck RSA SHA256:I8wESENSORED agent
debug1: Offering public key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug1: Server accepts key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED explicit agent
debug2: client_session2_setup: id 0
debug1: Sending command: echo did it
debug2: channel_input_status_confirm: type 99 id 0
did it
(dev3) 1 42356.....................................:Fri 05 Aug 2022 10:01:59 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> grep '\.pub' /tmp/ssh-strace 
1928608 openat(AT_FDCWD, "/home/yoh/.ssh/id_rsa.pub", O_RDONLY) = 4
1928608 openat(AT_FDCWD, "/home/yoh/.ssh/id_rsa-cert.pub", O_RDONLY) = -1 ENOENT (No such file or directory)
(dev3) 1 42357.....................................:Fri 05 Aug 2022 10:02:08 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> mv /home/yoh/.ssh/id_rsa.pub{,.aside}
(dev3) 1 42358.....................................:Fri 05 Aug 2022 10:02:31 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> ssh -p 22 127.0.0.1 echo did it
did it
(dev3) 1 42359.....................................:Fri 05 Aug 2022 10:02:41 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> strace -f -o /tmp/ssh-strace ssh -vvv -p 22 127.0.0.1 echo did it 2>&1 | grep id
debug1: identity file /home/yoh/.ssh/id_rsa type -1
debug1: identity file /home/yoh/.ssh/id_rsa-cert type -1
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 5 keys
debug1: Will attempt key: /home/yoh/.ssh/id_rsa_mailcheck RSA SHA256:I8wESENSORED agent
debug1: Will attempt key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED agent
debug1: Will attempt key: /home/yoh/.ssh/id_rsa  explicit
debug1: Offering public key: /home/yoh/.ssh/id_rsa_mailcheck RSA SHA256:I8wESENSORED agent
debug1: Offering public key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED agent
debug1: Server accepts key: /home/yoh/.ssh/id_rsa RSA SHA256:bFbbSENSORED agent
debug2: client_session2_setup: id 0
debug1: Sending command: echo did it
debug2: channel_input_status_confirm: type 99 id 0
did it
(dev3) 1 42360.....................................:Fri 05 Aug 2022 10:02:48 AM EDT:.
(git)lena:~/proj/misc/asyncssh[develop]git-annex
$> grep '\.pub' /tmp/ssh-strace                                                    
1929014 openat(AT_FDCWD, "/home/yoh/.ssh/id_rsa.pub", O_RDONLY) = -1 ENOENT (No such file or directory)
1929014 openat(AT_FDCWD, "/home/yoh/.ssh/id_rsa-cert.pub", O_RDONLY) = -1 ENOENT (No such file or directory)

I have following keys loaded

2048 SHA256:I8wESENSORED /home/yoh/.ssh/id_rsa_mailcheck (RSA)
2048 SHA256:bFbbSENSORED /home/yoh/.ssh/id_rsa (RSA)
3072 SHA256:5XvGSENSORED mih@meiner (RSA)
2048 SHA256:VB+jSENSORED annex (RSA)
2048 SHA256:aGl5SENSORED yoh@novo (RSA)

although I don't know how it gets the signature if .pub is not present since at least ssh-keygen seems to operate on .pub

$>  ssh-keygen -l -f  .ssh/id_rsa
.ssh/id_rsa is not a key file.

$>  ssh-keygen -l -f .ssh/id_rsa.pub.aside |  sed -e 's,\(SHA256:....\)\([^ ]*\),\1SENSORED,g'
2048 SHA256:bFbbSENSORED yoh@novo (RSA)

[ronf]

Yeah - without a .pub file, it would still be able to load the key, but it might not be able to tell that this key is the same as one of the agent keys, and assuming the private key file was encrypted with a passphrase, I would expect it to need you to enter that passphrase before it could use the key (when not getting it through the agent).

I was above to confirm that I see different behavior in OpenSSH with and without the ".pub" file present, for older private key types like PEM. With OpenSSH's private key format, I see the same behavior with and without the ".pub" file, as OpenSSH's private key format contains both the public and private keys and no longer needs a separate ".pub" file.

In cases where a public key is available, it looks like OpenSSH is smart enough to skip trying to decrypt a key specified via IdentityFile in .ssh/config in cases where the key was already loaded into the agent (determined by comparing the public key data). So, it could still end up prompting for passphrase, but only for encrypted private keys listed in .ssh/config which were not already loaded into the agent.

I'm not sure how easy it would be to completely replicate this behavior in AsyncSSH, but I'll take a look. I may just fall back to always ignoring encrypted keys in the config file when no passphrase is provided, similar to how it automatically ignores encrypted private keys in the .ssh directory matching the default names.

[ronf]

Trying to match OpenSSH's behavior exactly would be a very significant change, since right now the key loading happens completely separate from looking up SSH agent keys, and there's currently no code which tries to determine the public version of a key when detecting it is encrypted, to decide whether to load it or not.

Instead, I'm looking to add the following patch:

diff --git a/asyncssh/connection.py b/asyncssh/connection.py
index 89a82b2..ea98a23 100644
--- a/asyncssh/connection.py
+++ b/asyncssh/connection.py
@@ -6870,7 +6870,7 @@ class SSHClientConnectionOptions(SSHConnectionOptions):
                 client_keys: _ClientKeysArg = (),
                 client_certs: Sequence[FilePath] = (),
                 passphrase: Optional[BytesOrStr] = None,
-                ignore_encrypted: bool = False,
+                ignore_encrypted: bool = True,
                 gss_host: DefTuple[Optional[str]] = (),
                 gss_kex: DefTuple[bool] = (), gss_auth: DefTuple[bool] = (),
                 gss_delegate_creds: DefTuple[bool] = (),
@@ -7040,6 +7040,8 @@ class SSHClientConnectionOptions(SSHConnectionOptions):

         if client_keys == ():
             client_keys = cast(_ClientKeysArg, config.get('IdentityFile', ()))
+        else:
+            ignore_encrypted = False

         if client_certs == ():
             client_certs = \

This changes the default for the ignore_unencrypted argument to be True, but sets this argument to False when keys are specified via the client_keys argument rather than through a config file. The existing documentation suggests that this option is only meant to apply in the config file case, and this change actually makes the code better match the documentation. You can explicitly set it to False to force errors for unencrypted keys in config files, but regardless of what you set it to, specifying encrypted keys in client_keys without a passphrase will always raise an error, regardless of what ignore_encrypted is set to.

This change should allow your examples to run without raising the exception about needing a passphrase, similar to what you get today with OpenSSH.

[ronf]

After some thought, I went with a slightly different approach which more closely preserves current behavior, where the caller can override ignore_encrypted for either keys loaded from the config file or via client_keys. However, when not specified, the default is now to ignore unencrypted keys in config files case when no passphrase is specified. When using client_keys, the previous behavior of raising an error by default remains in place.

See commit 004799e for the detailed change.

[ronf]

This change is now available in AsyncSSH 2.12.0.

[yarikoptic]

Thank you, appreciated ! FWIW confirming that the fix seems to work for me ;)

[ronf]

Great - thanks for letting me know!