Oracle VBS (Git host) doesn't work with Flux anymore due to SSH key type

[ajhindle]

Describe the bug

I can't run Flux reconcile against my repo in Oracle Visual Builder Studio (Git host) anymore. Before September 2023 it worked fine, but in the VBS latest update I think they made it so only SHA2 RSA keys work now. VBS don't support ECDSA keys, only RSA.

flux reconcile kustomization flux-system --with-source
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git': ssh: handshake failed: ssh: no common algorithm for host key; client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]'

Please note the message: "client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]"

Steps to reproduce

1. Delete existing Flux key.
2. Create new Flux key:
   flux create secret git flux-system --url=ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git --ssh-key-algorithm rsa --ssh-rsa-bits=4096
3. Reconcile: flux reconcile kustomization flux-system --with-source

Expected behavior

Expect the reconciliation to work without key handshake issues.

Screenshots and recordings

No response

OS / Distro

Ubuntu 20.04

Flux version

v2.1.1

Flux check

► checking prerequisites ✔ Kubernetes 1.27.4-eks-2d98532 >=1.25.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.35.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.1 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

Oracle Visual Builder Studio (VBS) in Oracle Cloud (OCI)

Container Registry provider

No response

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ajhindle]

Here is the known_host key from the flux-system secret:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=

[hiddeco]

Can you share the output of ssh-keyscan for that repository URL?

[ajhindle]

Ok - like this?

ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0

[makkes]

Your host only sends you an RSA host key but doesn't present that during the SSH handshake. Are you able to clone repos manually from that host?

[ajhindle]

Yes, I can clone manually with other SSH keys I've made.

[ajhindle]

Also, the SSH key that Flux created and added to the flux-system secret also works fine for a manual clone.

[stefanprodan]

Grep the known_hosts file on your machine and see what fingerprints are in there for that host. I guess you have SHA-2 keys in there.

[makkes]

Judging by all the info available now I'm pretty sure it's a server misconfiguration that you can mitigate by manually updating the flux-system Secret with the correct host key that you should be able to gather from your local known_hosts file as Stefan suggested.

[ajhindle]

The only differences between my local known_hosts and the known_hosts Flux made in the secret are:

• in local: the host is hashed
• in k8s flux secret: the host is plain text

The SSH key is the same in both.

The local's hashed host value won't work in the secret, right? I tried it anyway - no luck.

In Oracle VBS, when running Git commands with SSH, you seem to need to put the user name in also (see doco) - maybe that's part of the issue?
The flux secret doesn't know who the user is, AFAIK

[ajhindle]

Stefan asked me to run ssh-keyscan in a test pod (running linuxserver/openssh-server) on the EKS cluster running Flux. Did that, got the same result as when I run it locally.:

debug-shell:/# ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0

I also ran ssh -Q key from the test pod and got this output:

ssh-ed25519
[email protected]
[email protected]
[email protected]
ecdsa-sha2-nistp256
[email protected]
ecdsa-sha2-nistp384
[email protected]
ecdsa-sha2-nistp521
[email protected]
[email protected]
[email protected]
ssh-dss
[email protected]
ssh-rsa
[email protected]

[ajhindle]

What command can I use to determine if the key is SHA1 or SHA2 ?

RE: what I wrote before "The flux secret doesn't know who the user is, AFAIK" I realised that the user name is stored in the "GitRepository" component, field "URL" i.e. ssh://username@repository

[ajhindle]

The Oracle team asks "which SSH library is Flux running? Is it OpenSSH?"

[stefanprodan]

We use https://github.com/go-git/go-git

[ajhindle]

Hi.

Today I have:

1. Run git clone from Oracle VBS Git, from the same EKS cluster in a bitnami/git pod - the clone works fine with GIT_SSH_COMMAND="ssh -i ~/.ssh/prv.key" git clone ssh://[email protected]/vbs-mel_continuous-deployment_343/cd-bfs.git.
2. Run flux create secret git with a regular RSA SHA2 key - this creates a secret in the EKS cluster with private key, public key and known_hosts data. The known_hosts section is not hashed. This produces the same error in the original issue description. I also tried replacing the known_hosts data with the hashed code from (1) - nothing changed, same end result.

Do we still think it's a server misconfiguration?

Is there a way to alter the SSH config file that Flux uses?

[ajhindle]

This issue appears to have been resolved by Oracle, I guess?
I can run bootstrap and reconcile with VBS Git fine now. When Flux couldn't SSH to VBS Git previously, the ssh-keyscan command returned fewer lines, fewer keys - see below results compared to previous output.

ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAG0aWG2uzvzrkfs0PX4LKquwUjX6zZJ0bSNWv8x2eQfl37/FufIDKn3CiHG4B62dJaifTqIZ0rzCj+kjqU4yJbNBwCjJUdlYykkRGN3Zx8Nhtlft7cDUWP0EBEMINUiRt5YVfio7A0vPXIy7mSsk3K45C/HFhZdUpI0WS6NqlIlnX65YA==
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP8N61DYwPVAIohNDIeI/fT+6d/C+V81ErnEnmr+qFa6