"The public key is of type ssh-rsa, not a certificate."

[roy-work]

Extension name (the extension in question)

az ssh

Description of issue (in as much detail as possible)

When I attempt to az ssh vm -g $A_RG -n $A_SERVER, I get,

» az ssh vm -g [snip RG name] -n [snip VM name]
[snip email]@[snip IP]: Permission denied (publickey).

We do not think this is a true positive authentication error:

• I've double checked that I am logged into the correct tenant (which is evident by the fact that az ssh vm attempted an SSH at all: it was able to look up the IP, and this particular VM is uniquely named among all VMs in all tenants I have access to)
• I've double-checked that I have the appropriate role assignment on the VM. Specifically, I have the role "Virtual Machine Administrator Login" assigned to me.
• I've successfully SSH'd into this VM in that past, and to my knowledge, my access has not changed. (And indeed, my check above confirms that.)

/var/log/auth.log on the server, when I attempt a log in, says,

Oct 26 21:40:04 [snip]-server aad_certhandler[12861]: Version: 1.0.015950001; user: [snip, my email]
Oct 26 21:40:04 [snip]-server aad_certhandler[12861]: The public key is of type ssh-rsa, not a certificate.

If I replace aad_certhandler with a small wrapper that simply logs calls, it indeed appears to be true that the only thing it gets invoked appears to be a 3072-bit ssh-rsa pubkey. I presume this is the key that az ssh vm autogenerates.

Normally, and when I have a co-worker attempt a log in, it appears to get invoked with an SSH certificate, and sometimes, a pubkey. I presume it is the invocation with an SSH cert that leads, eventually, to a successful login.

It is not clear to us why, when I run az ssh vm, it does not seem to lead to a certificate being sent.

Additionally, if I run az ssh config, & then ssh -F <resulting config file> <that VM> I do get,

debug1: Will attempt key: /var/folders/gk/[snip]/T/aadsshcertdotd07nv/id_rsa.pub-aadcert.pub RSA-CERT SHA256:[a hash?] explicit

However, I also get,

debug1: Offering public key: /var/folders/gk/[snip]/T/aadsshcertdotd07nv/id_rsa.pub-aadcert.pub RSA-CERT SHA256:[a hash?] explicit
debug1: send_pubkey_test: no mutual signature algorithm

And that Seems Bad. But it is not clear how there would not be a mutual signature algorithm for me, but there would be for my co-worker? (I've not yet had him replicate this trial; when I do, I'll adjust this to note the difference in our logs.)

I've also temporarily mv ~/.ssh/config ~/.ssh/config.out-of-the-way, to ensure that there isn't some cross-talk. My assumption is that my personal config is not used, as az will pass -F to ssh when it execs it. Moving my config aside had no effect.

(We also have an internal support ticket with Azure to this effect.)

[yonzhan]

route to service team

[roy-work]

TL;DR: Azure needs to test AADLogin w/ OpenSSH 8.8p1; 8.8p1 disables SHA-1, and this appears to break AADLogin, which relies on it. Azure needs to fix this by issuing SSH certs in a format that uses modern, non-broken cryptography. I.e., not [email protected]. Preferably [email protected], but [email protected] or [email protected] would perhaps also work.

The key different appears to be that I'm on a modern version of OpenSSH. Specifically, I'm on 8.8p1; my coworker is 8.1p1; I've now tested, and I can SSH with 8.7p1, but not 8.8p1. This, plus the "no mutual signature algorithm" error, pretty heavily indicates that this passage from the release notes is the problem:

This release disables RSA signatures using the SHA-1 hash algorithm by default. This change has been made as the SHA-1 hash algorithm is cryptographically broken

Now, Azure's SSH certs appear to be signed using SHA-2. However, it seems like the problem is that they're of the type [email protected]; when I connect with 8.8p1, this isn't in the list of server signature algorithms that SSH is considering.

I've patched ssh, both 8.7p1 & 8.8p1, locally, with the following:

--- sshconnect2.c	2021-10-26 19:03:50.000000000 -0400
+++ sshconnect2.c	2021-10-27 10:51:17.000000000 -0400
@@ -1192,9 +1192,6 @@
 	    (key->type != KEY_RSA && key->type != KEY_RSA_CERT) ||
 	    (key->type == KEY_RSA_CERT && (ssh->compat & SSH_BUG_SIGTYPE))) {
 		/* Filter base key signature alg against our configuration */
+		debug_f("no-msa: returning from first if");
+		debug_f("no-msa: match_list: client = %s", sshkey_ssh_name(key));
+		debug_f("no-msa: match_list: server = %s", options.pubkey_accepted_algos);
 		return match_list(sshkey_ssh_name(key),
 		    options.pubkey_accepted_algos, NULL);
 	}

This seems to be where we're figuring out what signature alg. to use; a failure here seems to net me the "no mutual signature algorithm".

If I patch 8.7p1:

debug1: Offering public key: /var/folders/[snip]/id_rsa.pub-aadcert.pub RSA-CERT SHA256:[hash] explicit
debug1: key_sig_algorithm: no-msa: returning from first if
debug1: key_sig_algorithm: no-msa: match_list: client = [email protected]
debug1: key_sig_algorithm: no-msa: match_list: server = [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: [snip]

& things go on to a successful login.

If I patch 8.8p1:

debug1: Offering public key: /var/folders/gk/[snip]/id_rsa.pub-aadcert.pub RSA-CERT SHA256:[hash] explicit
debug1: key_sig_algorithm: no-msa: returning from first if
debug1: key_sig_algorithm: no-msa: match_list: client = [email protected]
debug1: key_sig_algorithm: no-msa: match_list: server = [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug1: send_pubkey_test: no mutual signature algorithm

& things fail.

It is hard to see w/ how Github won't wrap those code blocks, but [email protected] appears in server for 8.7p1, and it does not in 8.8p1.

My best guess here is that SSH is going to not only want to present the certificate, but also prove that it holds the private key corresponding to the public key listed on the cert. To do that, it'll generate a signature, and the [email protected] scheme presumes SHA-1. N.b., this is separate from the CA signature on the cert itself. (Which does appear to be SHA-2.) Following the presumption, because [email protected] uses SHA-1, ssh in 8.8p1 removes it from the list of server-acceptable protocols, as it is insecure.

If I'm reading PROTOCOL.certkeys correctly, 8.8p1 really wants a [email protected] cert, and that would use modern crypto.

The release notes give us one more clue/hint/thing:

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded or have not closely tracked improvements in the SSH protocol. For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options. For example, the following stanza in ~/.ssh/config will enable RSA/SHA1 for host and user authentication for a single destination host:

Host old-host
   HostkeyAlgorithms +ssh-rsa
   PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy implementations can be upgraded or reconfigured with another key type (such as ECDSA or Ed25519).

This doesn't work here, as we're not using ssh-rsa. What does work is PubkeyAcceptedKeyTypes [email protected]. Now, due to https://github.com/Azure/azure-cli/issues/19101, specifying that option is tricky; az ssh vm is essentially useless; one has to make the config with az ssh config, and then invoke ssh manually.

What it boils down to is: test Azure AAD login with 8.8p1; you will find it broken.

[ernstp]

I think I'm also running into this when using az vm ssh from Ubuntu 22.04. The error message I get is:

The OpenSSH server version in the target VM 7.6 is too old. Version incompatible with OpenSSH client version 8.9. Refer to https://bugzilla.mindrot.org/show_bug.cgi?id=3351 for more information.

PubkeyAcceptedAlgorithms [email protected] to ssh config solves it yes.

[AchilleAsh]

You can now also pass options to ssh when using az ssh vm:

az ssh vm -n myvm -g mygroup -- -o [email protected]

[harshavmb]

I upgraded my Mac OS to Ventura (13.2) which bumped openssh to 9.0 (don't know the previous version) & started seeing this error while connecting to RHEL7 VMs.

Having checked openssh rpm versions, the maximum I could see is 7.4pl for RHEL7. The workaround -- -o [email protected] needs to be selectively applied for RHEL7 VMs which breaks automation workloads.

I do expect it get worse when openssh versions are bumped to more than 8.7 for 5.x kernel based OS like RHEL9.

Is this something possible to get this fixed by az-cli?

[eranikid]

Passing -o [email protected] has stopped working for me, I'm again getting an

The OpenSSH server version in the target VM 7.4 is too old. Version incompatible with OpenSSH client version 9.2. Refer to https://bugzilla.mindrot.org/show_bug.cgi?id=3351 for more information.

Any updated workarounds?

[GDMTT]

I'm having the same issue, this is my first time using the az ssh

I'm getting the following error : PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az ssh vm --name $serverName --resource-group $ResourceGroup --subscription $Subscription --verbose --prefer-private-ip OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5 [email protected]@10.203.1.13: Permission denied (publickey). The OpenSSH client version 7.7 is too old. Version incompatible with OpenSSH server version 8.9 in the target VM. Refer to https://bugzilla.mindrot.org/show_bug.cgi?id=3351 for more information. [32mCommand ran in 3.245 seconds (init: 0.407, invoke: 2.839)[0m

I have tried the workarounds presented in https://learn.microsoft.com/en-us/azure/active-directory/devices/howto-vm-sign-in-azure-ad-linux#getting-permission-denied-when-trying-to-connect-from-azure-shell-to-linux-red-hatoraclecentos-7x-vm

az ssh vm -n myVM -g MyResourceGroup -- -A -o "PubkeyAcceptedKeyTypes= [email protected]"

and get the same results :

PS C:\Users\srv_gregorym.ssh> az ssh vm --name $serverName --resource-group $ResourceGroup --subscription $Subscription --verbose --prefer-private-ip -- -A -o "PubkeyAcceptedKeyTypes= [email protected]" OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5 [email protected]@10.203.1.13: Permission denied (publickey). The OpenSSH client version 7.7 is too old. Version incompatible with OpenSSH server version 8.9 in the target VM. Refer to https://bugzilla.mindrot.org/show_bug.cgi?id=3351 for more information. [32mCommand ran in 3.773 seconds (init: 0.446, invoke: 3.326)[0m

PS C:\Users\srv_gregorym.ssh>

Version information: ####################################################################### PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az --version azure-cli 2.51.0

core 2.51.0 telemetry 1.1.0

Extensions: ssh 2.0.1

Dependencies: msal 1.24.0b1 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\srv_gregorym.azure\cliextensions'

Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date. PS C:\Program Files\Microsoft SDKs\Azure\CLI2>

####################################################################### PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az extension list -otable Experimental ExtensionType Name Path Preview Version

---

False whl ssh C:\Users\srv_gregorym.azure\cliextensions\ssh False 2.0.1 PS C:\Program Files\Microsoft SDKs\Azure\CLI2> ####################################################################### PS C:\Program Files\Microsoft SDKs\Azure\CLI2> $PSVersionTable

Name Value

---

PSVersion 5.1.17763.3770 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.17763.3770 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1

PS C:\Program Files\Microsoft SDKs\Azure\CLI2> ####################################################################### Windows Server 2019

Is there an eta on this, I dont want to hand the server over with the users using local logon account.
I cant see a stable release installer for OpenSSH to upgrade the local OpenSSH