flux2
flux2 copied to clipboard
Oracle VBS (Git host) doesn't work with Flux anymore due to SSH key type
Describe the bug
I can't run Flux reconcile against my repo in Oracle Visual Builder Studio (Git host) anymore. Before September 2023 it worked fine, but in the VBS latest update I think they made it so only SHA2 RSA keys work now. VBS don't support ECDSA keys, only RSA.
flux reconcile kustomization flux-system --with-source
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git': ssh: handshake failed: ssh: no common algorithm for host key; client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]'
Please note the message: "client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]"
Steps to reproduce
- Delete existing Flux key.
- Create new Flux key:
flux create secret git flux-system --url=ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git --ssh-key-algorithm rsa --ssh-rsa-bits=4096
- Reconcile:
flux reconcile kustomization flux-system --with-source
Expected behavior
Expect the reconciliation to work without key handshake issues.
Screenshots and recordings
No response
OS / Distro
Ubuntu 20.04
Flux version
v2.1.1
Flux check
► checking prerequisites ✔ Kubernetes 1.27.4-eks-2d98532 >=1.25.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.35.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.1 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed
Git provider
Oracle Visual Builder Studio (VBS) in Oracle Cloud (OCI)
Container Registry provider
No response
Additional context
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Here is the known_host key from the flux-system secret:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
Can you share the output of ssh-keyscan
for that repository URL?
Ok - like this?
ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
Your host only sends you an RSA host key but doesn't present that during the SSH handshake. Are you able to clone repos manually from that host?
Yes, I can clone manually with other SSH keys I've made.
Also, the SSH key that Flux created and added to the flux-system secret also works fine for a manual clone.
Grep the known_hosts file on your machine and see what fingerprints are in there for that host. I guess you have SHA-2 keys in there.
Judging by all the info available now I'm pretty sure it's a server misconfiguration that you can mitigate by manually updating the flux-system
Secret with the correct host key that you should be able to gather from your local known_hosts file as Stefan suggested.
The only differences between my local known_hosts and the known_hosts Flux made in the secret are:
- in local: the host is hashed
- in k8s flux secret: the host is plain text
The SSH key is the same in both.
The local's hashed host value won't work in the secret, right? I tried it anyway - no luck.
In Oracle VBS, when running Git commands with SSH, you seem to need to put the user name in also (see doco) - maybe that's part of the issue?
The flux secret doesn't know who the user is, AFAIK
Stefan asked me to run ssh-keyscan
in a test pod (running linuxserver/openssh-server
) on the EKS cluster running Flux.
Did that, got the same result as when I run it locally.:
debug-shell:/# ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
I also ran ssh -Q key
from the test pod and got this output:
ssh-ed25519
[email protected]
[email protected]
[email protected]
ecdsa-sha2-nistp256
[email protected]
ecdsa-sha2-nistp384
[email protected]
ecdsa-sha2-nistp521
[email protected]
[email protected]
[email protected]
ssh-dss
[email protected]
ssh-rsa
[email protected]
What command can I use to determine if the key is SHA1 or SHA2 ?
RE: what I wrote before "The flux secret doesn't know who the user is, AFAIK" I realised that the user name is stored in the "GitRepository" component, field "URL" i.e. ssh://username@repository
The Oracle team asks "which SSH library is Flux running? Is it OpenSSH?"
We use https://github.com/go-git/go-git
Hi.
Today I have:
- Run
git clone
from Oracle VBS Git, from the same EKS cluster in abitnami/git
pod - the clone works fine withGIT_SSH_COMMAND="ssh -i ~/.ssh/prv.key" git clone ssh://[email protected]/vbs-mel_continuous-deployment_343/cd-bfs.git
. - Run
flux create secret git
with a regular RSA SHA2 key - this creates a secret in the EKS cluster with private key, public key and known_hosts data. The known_hosts section is not hashed. This produces the same error in the original issue description. I also tried replacing the known_hosts data with the hashed code from (1) - nothing changed, same end result.
Do we still think it's a server misconfiguration?
Is there a way to alter the SSH config file that Flux uses?
This issue appears to have been resolved by Oracle, I guess?
I can run bootstrap and reconcile with VBS Git fine now.
When Flux couldn't SSH to VBS Git previously, the ssh-keyscan
command returned fewer lines, fewer keys - see below results compared to previous output.
ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAG0aWG2uzvzrkfs0PX4LKquwUjX6zZJ0bSNWv8x2eQfl37/FufIDKn3CiHG4B62dJaifTqIZ0rzCj+kjqU4yJbNBwCjJUdlYykkRGN3Zx8Nhtlft7cDUWP0EBEMINUiRt5YVfio7A0vPXIy7mSsk3K45C/HFhZdUpI0WS6NqlIlnX65YA==
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP8N61DYwPVAIohNDIeI/fT+6d/C+V81ErnEnmr+qFa6