tg123
Finding a way to not return "partial success" during authentication
Hi. Thanks again for getting the dedup banner feature created. It has helped our use case.
We spent time understanding the protocol flow between downstream, sshpiper, and upstream. We have a much better understanding of what has been occurring in our configuration.
It appears that sshpiper returns a "partial success" message to the downstream client in response to a failed authentication against an upstream. It appears to do this as a way to hunt for a successful connection with an upstream server, since each authn request from the client will initiate a distinct connection from sshpiper to a (possibly different) upstream.
We assume this is the mechanism sshpiper uses to find a potential upstream server with which to connect in a load balancing scenario. That is, keep trying until a connection between sshpiper and an upstream is established. This assumes an upstream may reject connections during authn due to load issues and use authn rejection with partial success to communicate "try again".
Because sshpiper responds with partial success a client like OpenSSH tries to repeat the auth attempt. This spawns a new sshpiper-to-upstream connection. Assuming these connections spread across multiple upstreams it may eventually find an upstream that accepts the connection, thereby load balancing connections.
We observed the ssh connections between sshpiper and the upstream aren't actually closed, And they are simply not reused. Each auth attempt initiated by the downstream client garners a unique connection to the upstream from sshpiper. We assume some garbage handling eventually closes these abandoned connections.
Here's an example of the ssh -vvv ouput when connecting via sshpiper to an upstream host using password authn. In our case, our upstream recognizes the account is expired and issues a input_userauth_banner. A direct connect to such a server would terminate at this point after packet: type 53 message and the servers termination of the connection (see below). It seems sshpiper issues a partial success which causes the downstream client to try again.
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
user@host's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 53
debug3: input_userauth_banner: entering
Your account has expired; please contact your system administrator
account expired 1 days ago
debug3: receive packet: type 51
Authenticated using "password" with partial success.
debug1: Authentications that can continue: password,publickey
Permission denied, please try again.
user@host's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
Authenticated using "password" with partial success.
debug1: Authentications that can continue: password,publickey
Permission denied, please try again.
user@host's password:
While we would prefer the connection to terminate after the input_user_auth_banner and the packet: type 51 (SSH_MSG_USERAUTH_FAILURE), it seems somewhat harmless in a password-based authn scenario. The user would not be surprised by a second password prompt since they might have typed their password incorrectly. They could try again or see the message and abandon the connection.
When key-based authn is used, however, pushing this partial success back to the downstream client simply causes it to retry the connection with the same keypair. Note that this key pair either works or it doesn't. There is no careful retyping by the user. By pushing the partial success back to the downstream with key-based authn the down stream server will just keep retrying the authentication until some other mechanism succeeds or cuts off the attempt, e.g. the failtoban plugin. This does support sshpiper's hunt for an upstream to take the connection for load balancing but isn't effective is the account is universally rejected because of expiration or other permanent state.
Here's the same example as above but with key based authn. You can see the banner message is displayed but then the connection continues with a partial success message.
debug2: pubkey_prepare: done
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:XXXXXXX
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 53
debug3: input_userauth_banner: entering
Your account has expired; please contact your system administrator
account expired 1 days ago
debug3: receive packet: type 60
debug1: Server accepts key: /home/user/.ssh/id_rsa RSA SHA256:XXXXXXX
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:XXXXXXX
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:XXXXXXX
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated using "publickey" with partial success.
After the partial success message is received by the downstream it retries the authn with the same public key. In our case, this will fail again because the users account remains expired, regardless of which upstream they communicate with.
Note: in our config it happens to point to the same upstream. It needn't and could load balance across others, but each upstream would reject the connection because the account is expired.
What we see instead is that this sequence is repeated until the max connection attempt is exceeded as configured with failtoban plugin. This leads to our downstream client being banned because of too many connection attempts. The downstream user has no way to control this client behavior. The client will just keep repeating the authn attempt because it interprets partial success as try again.
Here is the repeated attempt which continues until the allowed attempts are exceeded.
debug1: Authentications that can continue: password,publickey
<snip key scan>
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:XXXXXXX
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/user/.ssh/id_rsa RSA SHA256:XXXXXXX
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:XXXXXXX
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:XXXXXXX
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated using "publickey" with partial success.
Note: the input_userauth_banner is not repeated because we are now using the banner dedup feature.
In contrast, here is a an example of a password authn against an ordinary OpenSSH server without sshpiper proxying. It seems the client still reprompts the user for another password after an incorrect password and the packet: type 51 is returned.
I don't understand SSH well enough to understand the difference between this scenario and the sshpiper partial success approach above.
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
user@host's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
Permission denied, please try again.
user@hosts password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
Here is what a successful auth directly to the upstream without sshpiper looks like for an expired account. Notice that the connection is closed by the server, preventing further authn attempts. This is our preferred behavior.
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
user@host's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 53
debug3: input_userauth_banner: entering
Your account has expired; please contact your system administrator
account expired 1 days ago
Connection closed by XX.XX.49.178 port 22
If the partial success mechanism is needed for the sshpiper load balancing functionality, we'd like to explore how we might add a configuration option that that instructs sshpiper not to return partial success when the client uses key based authn. That would help address our sshpiper connection routing scenario where all upstreams share the same account state. That is, if one upstream rejects an expired account, all will. In this case, it would be correct for the downstream client not to retry key-based authn repeatedly.
I hope the above makes sense and is a correct interpretation of behavior and function. Let me know if clarifications are needed.
Thanks for your responsiveness, fixes, and continued suggestions.
We are still learning the code base and will continue to look for a place to contribute further.
