Too many certs in ssh-agent makes auth fail with 'too many authentication failures'

[fbomlisboa]

After generating several certificates and authenticating with gsh, the ssh-agent caches the certificates, and after a number of certificates new authentications fail when new certificates are issued.

I had a lot of certificates in gsh folder (not all of them were cached):

MacBook-Pro-37:prod felipe$ ll
total 272
drwxr-x---  36 felipe  staff   1.1K Apr 17 09:51 .
drwxr-x---   3 felipe  staff    96B Jul 26  2019 ..
-rw-------   1 felipe  staff   3.2K Apr 17 09:47 1V8tM1JuJBFfVq1U6zBndpjW5KrURlbe
-rw-r--r--   1 felipe  staff   2.4K Apr 17 09:47 1V8tM1JuJBFfVq1U6zBndpjW5KrURlbe-cert.pub
-rw-------   1 felipe  staff   3.2K Jul 26  2019 3dNWW2IEnHCNRpW8y0TtO5WGUYFdxYnm
-rw-r--r--   1 felipe  staff   2.4K Jul 26  2019 3dNWW2IEnHCNRpW8y0TtO5WGUYFdxYnm-cert.pub
-rw-------   1 felipe  staff   3.2K Jul 26  2019 3uwVNF3Edb3CjLGishLyqfNPxEJ65fj0
-rw-r--r--   1 felipe  staff   2.4K Jul 26  2019 3uwVNF3Edb3CjLGishLyqfNPxEJ65fj0-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 17 09:51 8kfKvAnDdRvcSXnYgpIypJsV4QOQrihB
-rw-r--r--   1 felipe  staff   2.4K Apr 17 09:51 8kfKvAnDdRvcSXnYgpIypJsV4QOQrihB-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 16 17:35 AeaBwc7QjqAjCl1pGViEGt4HjdZ6cngI
-rw-r--r--   1 felipe  staff   2.4K Apr 16 17:35 AeaBwc7QjqAjCl1pGViEGt4HjdZ6cngI-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 16 18:41 Jxvpytz6V3Hn9Hh4uawm7BoXh2FRxdP8
-rw-r--r--   1 felipe  staff   2.4K Apr 16 18:41 Jxvpytz6V3Hn9Hh4uawm7BoXh2FRxdP8-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 16 18:41 Lboi113d2M1hWlZYgv5YV8C8PxOzUYFO
-rw-r--r--   1 felipe  staff   2.4K Apr 16 18:41 Lboi113d2M1hWlZYgv5YV8C8PxOzUYFO-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 15 11:04 MwJ6HFB1BsVgQVuTC6pVh01CFlTe6ejh
-rw-r--r--   1 felipe  staff   2.4K Apr 15 11:04 MwJ6HFB1BsVgQVuTC6pVh01CFlTe6ejh-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 16 19:22 ZJEiVajueQfsTVt6HB8IN4fFh5DPxYPG
-rw-r--r--   1 felipe  staff   2.4K Apr 16 19:22 ZJEiVajueQfsTVt6HB8IN4fFh5DPxYPG-cert.pub
-rw-------   1 felipe  staff   3.2K Jul 26  2019 aS9DVHtOsnzOCZ0YPyCpqw1MSOHhRQP1
-rw-r--r--   1 felipe  staff   2.4K Jul 26  2019 aS9DVHtOsnzOCZ0YPyCpqw1MSOHhRQP1-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 15 11:37 eNyGTWf90Ilx9L4s97OrPpJlufGfHekk
-rw-r--r--   1 felipe  staff   2.4K Apr 15 11:37 eNyGTWf90Ilx9L4s97OrPpJlufGfHekk-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 17 09:49 gE2ALbq0nUCw8NBdnoFoqLfxxvS1YIK8
-rw-r--r--   1 felipe  staff   2.4K Apr 17 09:49 gE2ALbq0nUCw8NBdnoFoqLfxxvS1YIK8-cert.pub
-rw-------   1 felipe  staff   3.2K Jul 26  2019 jthdwxmMw6ouPL2H982K1L1AqriVoTws
-rw-r--r--   1 felipe  staff   2.4K Jul 26  2019 jthdwxmMw6ouPL2H982K1L1AqriVoTws-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 17 09:48 lOM5VORINe3sH0sb1PcMtTaJlLFLfkp7
-rw-r--r--   1 felipe  staff   2.4K Apr 17 09:48 lOM5VORINe3sH0sb1PcMtTaJlLFLfkp7-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 17 09:47 rB7KB6J3Czp3ZmRSThMTvVL0FTbocFCk
-rw-r--r--   1 felipe  staff   2.4K Apr 17 09:47 rB7KB6J3Czp3ZmRSThMTvVL0FTbocFCk-cert.pub
-rw-------   1 felipe  staff   3.2K Apr 16 19:08 tgJHdKcEwXleE51ZG81q9CE0mRtFvFTr
-rw-r--r--   1 felipe  staff   2.4K Apr 16 19:08 tgJHdKcEwXleE51ZG81q9CE0mRtFvFTr-cert.pub
-rw-------   1 felipe  staff   3.2K Jul 26  2019 xk0n5naXONX4EltF1qDmYw7Z9GUJX930
-rw-r--r--   1 felipe  staff   2.4K Jul 26  2019 xk0n5naXONX4EltF1qDmYw7Z9GUJX930-cert.pub

When trying a new authentication it would fail. I used the option -d to see which certificate was trying to use, and added -v to ssh command:

OpenSSH_8.2p1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /Users/felipe/.ssh/config
debug1: /Users/felipe/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to HOST_IP [HOST_IP] port 22.
debug1: Connection established.
debug1: identity file /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU type -1
debug1: identity file /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU-cert type 4
debug1: identity file /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU-cert.pub type 4
debug1: identity file /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU-cert.pub-cert type -1
debug1: identity file /Users/felipe/.ssh/id_rsa type 0
debug1: identity file /Users/felipe/.ssh/id_rsa-cert type 4
debug1: Local version string SSH-2.0-OpenSSH_8.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to HOST_IP:22 as 'USER'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:uZqx1/SL3Y7Q2Zm/qVrcivUJcWFR5diMMBGt+eXh2JQ
debug1: Host 'HOST_IP' is known and matches the ECDSA host key.
debug1: Found key in /Users/felipe/.ssh/known_hosts:902
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/felipe/.ssh/id_rsa RSA SHA256:eMb0q08Jp36LQGHmy8aEMWG5KVVdHo4WyfcB31Pcwus explicit agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/Jxvpytz6V3Hn9Hh4uawm7BoXh2FRxdP8 RSA SHA256:pM4+t6WXSHsP/caXWM+S2kvi8ApUG1c2UvOeUiJo1hM agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/Lboi113d2M1hWlZYgv5YV8C8PxOzUYFO RSA SHA256:0f0Hs+pEPKwSeLtgNIEZqI9V4afpNDUOaGmGs710kys agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/tgJHdKcEwXleE51ZG81q9CE0mRtFvFTr RSA SHA256:tmPrt/XHj0pe76QB/s53PBBxqKhsx69DXzfXF7BrEps agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/ZJEiVajueQfsTVt6HB8IN4fFh5DPxYPG RSA SHA256:mg+rCTp+DrMfAGp/8Qg8aBUDDekTZrTgEgzpQpFY9D4 agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/1V8tM1JuJBFfVq1U6zBndpjW5KrURlbe RSA SHA256:sBVD8rTjS8/JTkIlIv9GiniGwmsJLrtLsrB+VhZjm18 agent
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU  explicit
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU RSA-CERT SHA256:J6YY9HztjAsNamxvhJ4YSlz7mNi4j77hsUZPRJnRni4 explicit
debug1: Will attempt key: /Users/felipe/.gsh/certs/prod/rNzw4nQoDirwKp8u7Bzxbta6lXsNgJUU-cert.pub RSA-CERT SHA256:J6YY9HztjAsNamxvhJ4YSlz7mNi4j77hsUZPRJnRni4 explicit
debug1: Will attempt key: /Users/felipe/.ssh/id_rsa RSA-CERT SHA256:eMb0q08Jp36LQGHmy8aEMWG5KVVdHo4WyfcB31Pcwus explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/felipe/.ssh/id_rsa RSA SHA256:eMb0q08Jp36LQGHmy8aEMWG5KVVdHo4WyfcB31Pcwus explicit agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/felipe/.gsh/certs/prod/Jxvpytz6V3Hn9Hh4uawm7BoXh2FRxdP8 RSA SHA256:pM4+t6WXSHsP/caXWM+S2kvi8ApUG1c2UvOeUiJo1hM agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/felipe/.gsh/certs/prod/Lboi113d2M1hWlZYgv5YV8C8PxOzUYFO RSA SHA256:0f0Hs+pEPKwSeLtgNIEZqI9V4afpNDUOaGmGs710kys agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/felipe/.gsh/certs/prod/tgJHdKcEwXleE51ZG81q9CE0mRtFvFTr RSA SHA256:tmPrt/XHj0pe76QB/s53PBBxqKhsx69DXzfXF7BrEps agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/felipe/.gsh/certs/prod/ZJEiVajueQfsTVt6HB8IN4fFh5DPxYPG RSA SHA256:mg+rCTp+DrMfAGp/8Qg8aBUDDekTZrTgEgzpQpFY9D4 agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /Users/felipe/.gsh/certs/prod/1V8tM1JuJBFfVq1U6zBndpjW5KrURlbe RSA SHA256:sBVD8rTjS8/JTkIlIv9GiniGwmsJLrtLsrB+VhZjm18 agent
Received disconnect from HOST_IP port 22:2: **Too many authentication failures**

ssh-agent had some certificates:

MacBook-Pro-37:~ felipe$ ssh-add -l 
2048 SHA256:eMb0q08Jp36LQGHmy8aEMWG5KVVdHo4WyfcB31Pcwus /Users/felipe/.ssh/id_rsa (RSA)
4096 SHA256:pM4+t6WXSHsP/caXWM+S2kvi8ApUG1c2UvOeUiJo1hM /Users/felipe/.gsh/certs/prod/Jxvpytz6V3Hn9Hh4uawm7BoXh2FRxdP8 (RSA)
4096 SHA256:0f0Hs+pEPKwSeLtgNIEZqI9V4afpNDUOaGmGs710kys /Users/felipe/.gsh/certs/prod/Lboi113d2M1hWlZYgv5YV8C8PxOzUYFO (RSA)
4096 SHA256:tmPrt/XHj0pe76QB/s53PBBxqKhsx69DXzfXF7BrEps /Users/felipe/.gsh/certs/prod/tgJHdKcEwXleE51ZG81q9CE0mRtFvFTr (RSA)
4096 SHA256:mg+rCTp+DrMfAGp/8Qg8aBUDDekTZrTgEgzpQpFY9D4 /Users/felipe/.gsh/certs/prod/ZJEiVajueQfsTVt6HB8IN4fFh5DPxYPG (RSA)
4096 SHA256:sBVD8rTjS8/JTkIlIv9GiniGwmsJLrtLsrB+VhZjm18 /Users/felipe/.gsh/certs/prod/1V8tM1JuJBFfVq1U6zBndpjW5KrURlbe (RSA)

After deleting all of them with ssh-add -D I was able to authenticate again.

[mdjunior]

I did some research on this behavior and I believe we have two options. We can use the options -o IdentityAgent=none or -o IdentitiesOnly=yes when gsh-cli calls the ssh client.

IdentitiesOnly
Specifies that ssh(1) should only use the configured authentication identity and certificate files 
(either the default files, or those explicitly configured in the ssh_config files or passed on the 
ssh(1) command-line), even if ssh-agent(1) or a PKCS11Provider or SecurityKeyProvider offers 
more identities. The argument to this keyword must be yes or no (the default). This option is 
intended for situations where ssh-agent offers many different identities.

IdentityAgent
Specifies the UNIX-domain socket used to communicate with the authentication agent.
This option overrides the SSH_AUTH_SOCK environment variable and can be used to 
select a specific agent. Setting the socket name to none disables the use of an authentication 
agent. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be 
read from the SSH_AUTH_SOCK environment variable. Otherwise if the specified value 
begins with a ‘$’ character, then it will be treated as an environment variable containing 
the location of the socket.

Arguments to IdentityAgent may use the tilde syntax to refer to a user's home directory, 
the tokens described in the TOKENS section and environment variables as described in 
the ENVIRONMENT VARIABLES section.

Ref: https://man.openbsd.org/ssh_config#IdentitiesOnly

Analyzing the options, I think that the ideal would be IdentitiesOnly, since interfering with the agent can generate unwanted effects. WDYT?