source-controller
source-controller copied to clipboard
Github Enterprise SSH handshake failing
Describe the bug
We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:
k get gitrepositories -n flux-system
NAME URL AGE READY STATUS
flux2-sync ssh://git@foo/abc/def 64d False failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
With version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):
k get gitrepositories -n flux-system
NAME URL AGE READY STATUS
flux2-sync ssh://git@foo/abc/def 64d True Fetched revision: main/4afa5937b48b12a42c0739c13870c282826b3979
If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message
Steps to reproduce
We installed flux with the community helm charts without any relevant change to the values.yaml
Expected behavior
It is working again ^^
Screenshots and recordings
No response
OS / Distro
N/A
Flux version
N/A
Flux check
► checking prerequisites ✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used ✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.18.2 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.21.1 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.17.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.22.2 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.23.1 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.22.3 ✔ all checks passed
Git provider
GitHub Enterprise
Container Registry provider
No response
Additional context
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
It seems that the bug was introduced in v0.22.0 of the sourcecontroller. Should i open the bug there?
@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation
are you using?
Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of source-controller
and enabling the experimental transport to see whether the issue persists? Here are more information about it: https://github.com/fluxcd/source-controller/issues/636#issuecomment-1080789920
@steveizzle thank you for reporting this. Do you mind sharing what
gitImplementation
are you using?
I used go-git and tried now libgit2. With libgit2 it is working.
Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of
source-controller
and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)
Sure, i will give it a try and report back
Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of
source-controller
and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)
The error seems to be consistent.
As soon as i enable the experimental transport neither libgit2 nor go-git is working:
env:
- name: EXPERIMENTAL_GIT_TRANSPORT
value: "true"
- name: RUNTIME_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: ghcr.io/fluxcd/source-controller:v0.22.5
We found out that one repository is using an ED25519 Key which seems to be still working. Normally we use RSA keys.
Thanks for that. Do you mind sharing the errors you get when using libgit2
and the experimental transport? The pod logs may have some extra information that could be useful.
We are getting the same error suddenly after redeploying a test cluster that we deploy often (almost every day) and doesn't give any errors with Flux.
Is there any resolution for this, or just keep trying until it works?
I currently pinned an old version, but will spent some time on getting more debug Infos the next days
@steveizzle @campanellaf this may have to do with some of the crypto level settings that may have changed upstream. So to confirm, you are using RSA for your authentication keypair, right? What is the algorithm used for the known_host you are providing?
We have just released a RC with some improvements specially around the libgit2
implementation. Would you mind to also checking whether it fixes your problem?
ghcr.io/fluxcd/source-controller:rc-4b3e0f9a
The issue still exists on v0.30.2.
@pjbgf I tried your RC with the go-git
implementation, but no luck.
The algorithm we're using for known_hosts is ecdsa-sha2-nistp256
@pjbgf Regarding the rc, Unfortunately the rc didnt fix the problem:
This is a example of a git repository in my test cluster:
k get deploy -n flux-system -o wide
helm-controller 1/1 1 1 132d manager ghcr.io/fluxcd/helm-controller:v0.21.0 app=helm-controller
image-automation-controller 1/1 1 1 132d manager ghcr.io/fluxcd/image-automation-controller:rc-48bcca59 app=image-automation-controller
image-reflector-controller 1/1 1 1 132d manager ghcr.io/fluxcd/image-reflector-controller:v0.18.0 app=image-reflector-controller
kustomize-controller 1/1 1 1 132d manager ghcr.io/fluxcd/kustomize-controller:v0.25.0 app=kustomize-controller
notification-controller 1/1 1 1 132d manager ghcr.io/fluxcd/notification-controller:v0.23.5 app=notification-controller
source-controller 1/1 1 1 132d manager ghcr.io/fluxcd/source-controller:rc-4b3e0f9a app=source-controller
k get gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync -o yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
annotations:
meta.helm.sh/release-name: flux2-sync
meta.helm.sh/release-namespace: flux-system
reconcile.fluxcd.io/requestedAt: "2022-06-02T17:34:40.013251+02:00"
creationTimestamp: "2022-01-21T12:20:41Z"
finalizers:
- finalizers.fluxcd.io
generation: 9
labels:
app.kubernetes.io/instance: flux-system
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: flux
helm.sh/chart: flux2-sync-0.4.0
name: flux2-sync
namespace: flux-system
resourceVersion: "556712592"
uid: 8f76a6b4-cf42-49e1-af14-9608a1dc25c2
spec:
gitImplementation: libgit2
interval: 5m0s
ref:
branch: main
secretRef:
name: flux2-secret
timeout: 20s
url: ssh://git@SECRET_ENTERPRISE_URL
status:
conditions:
- lastTransitionTime: "2022-06-02T15:16:18Z"
message: no artifact for resource in storage
observedGeneration: 9
reason: NoArtifact
status: "True"
type: Reconciling
- lastTransitionTime: "2022-06-02T15:32:34Z"
message: 'failed to checkout and determine revision: unable to fetch-connect to
remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
no supported methods remain'
observedGeneration: 9
reason: GitOperationFailed
status: "False"
type: Ready
- lastTransitionTime: "2022-06-02T15:32:34Z"
message: 'failed to checkout and determine revision: unable to fetch-connect to
remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
no supported methods remain'
observedGeneration: 9
reason: GitOperationFailed
status: "True"
type: FetchFailed
lastHandledReconcileAt: "2022-06-02T17:34:40.013251+02:00"
observedGeneration: 8
I also checked out the algorithms for the known_hosts and got the following:
Working:
- gitrepo1:
- known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256;
- key: ssh-ed25519
- gitrepo2:
- known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
- key: ecdsa-sha2-nistp384
Not Working:
- gitrepo3:
- known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
- identity key: ssh-rsa
- gitrepo4:
- known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256
- identity key: ssh-rsa
So i would say its not related to the known_hosts.. And still everything is working when i switch the source controller back to v0.21.2
I did not set the Experimental env Variable in the test with the rc. Is this still necessary with the rc?
@steveizzle we have released source-controller yesterday with managed transport enabled by default. The official image is here: https://github.com/fluxcd/source-controller/pkgs/container/source-controller/23808902?tag=v0.25.0
Please test it with go-git and libgit2, no env var needed.
Unfortunately we have got the same problem with v0.25.0 :(
k get deploy -n flux-system source-controller -o yaml | grep -i image:
image: ghcr.io/fluxcd/source-controller:v0.25.0
k get pods -n flux-system source-controller-d987bdf4f-mt8c8
NAME READY STATUS RESTARTS AGE
source-controller-d987bdf4f-mt8c8 1/1 Running 0 23m
k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"libgit2"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched (no change)
flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to fetch-connect to remote 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'
k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"go-git"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched
flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'
I also let it ran with loglevel debug, but i think stacktrace doesnt help much:
{"level":"error","ts":"2022-06-09T12:45:48.671Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_URL.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_URL.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:204\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
@steveizzle the repositories that are not working, are all of them on the same Git server?
Also, I noticed that the failing repos are using ssh-rsa for Auth-N, can you confirm that is enabled on the server?
We have released another release candidate that further improves on the connection management and observability.
Can you please give it a try using --log-level=trace
?
ghcr.io/fluxcd/source-controller:rc-b877bc21
About the auth-N:
ssh -fNvv steveizzle@SECRET_ENTERPRISE_URL
.....
debug2: host key algorithms: [email protected],ecdsa-sha2-nistp256,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
...
That should be a yes. And the rsa key is still working with the old version of the source controller..
Regarding the new rc:
k get deploy -n flux-system source-controller -o wide
source-controller 1/1 1 1 142d manager ghcr.io/fluxcd/source-controller:rc-b877bc21 app=source-controller
# ecdsa-sha2-nistp384:
k apply -f flux-secret-new.yaml
flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022
# ssh-rsa 4096:
k delete -f flux-secret-new.yaml
k apply -f flux-secret-new-rsa.yaml
flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to list remote for 'ssh://git@'SECRET_ENTERPRISE_URL: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'
# Relevant Reconcilliation Loop errors:
k logs -n flux-system source-controller-67f4567c-7bz5z
{"level":"error","ts":"2022-06-13T08:36:38.301Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","cid":"83666aeb-96e2-4a90-aaf1-6b3abe5f5bb1","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2022-06-13T08:36:38.301Z","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","name":"flux-app-test","namespace":"testing-cmo-flux","reconciler kind":"GitRepository","annotations":null,"error":"GitOperationFailed","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:113\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.recordEvent\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:123\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:80\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-06-13T08:36:38.302Z","logger":"events","msg":"Warning","object":{"kind":"GitRepository","namespace":"testing-cmo-flux","name":"flux-app-test","uid":"7e4d321f-b7b9-4370-bd92-70f825d88f42","apiVersion":"source.toolkit.fluxcd.io/v1beta2","resourceVersion":"567995259"},"reason":"GitOperationFailed","message":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"}
{"level":"error","ts":"2022-06-13T08:36:38.306Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
# v0.21.2:
k edit deploy -n flux-system source-controller
deployment.apps/source-controller edited
k get deploy -n flux-system source-controller -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
source-controller 1/1 1 1 142d manager ghcr.io/fluxcd/source-controller:v0.21.2 app=source-controller
flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022
i meet same issue. My flux env :
flux check
► checking prerequisites
✗ flux 0.24.1 <0.31.1 (new version is available, please upgrade)
✔ Kubernetes 1.21.9 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.12.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.15.5
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.16.0
✔ all checks passed
kubectl get gitrepositories -n flux-system
NAME URL READY STATUS AGE
flux-system ssh://[email protected]/xxxx/xxx-xxx.git False unable to clone 'ssh://[email protected]/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 98d
flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'unable to clone 'ssh://[email protected]/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'
@Kevinwoolworth @steveizzle would it be possible for you to post the description of the secret that the GitRepository
object refers to by running kubectl describe -n <your-namespace> secrets <your-secret>
? Thanks
@aryan9600 Sure, we switched to working authentications, but i surely will help to assist on this issue:
Name: flux-app-test
Namespace: testing-cmo-flux
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
identity: 2602 bytes
identity.pub: 571 bytes
known_hosts: 581 bytes
Hi is there any update or potential timeline for this issue?
Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue: @aryan9600 @stefanprodan
❯ k -n flux-system describe secret flux-system
Name: flux-system
Namespace: flux-system
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
identity: 306 bytes
identity.pub: 205 bytes
known_hosts: 171 bytes
❯ k get gitrepository -A
NAMESPACE NAME URL AGE READY STATUS
flux-system flux-system ssh://[email protected]/gitops/k8s-configs 139d False failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
flux-system gilogging https://gitlab.example.com/gi/apps/gilogging.git 139d True stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'
❯ flux check
► checking prerequisites
✗ flux 0.31.5 <0.32.0 (new version is available, please upgrade)
✔ Kubernetes 1.23.7-eks-4721010 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.11
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed
❯ flux get sources git
NAME REVISION SUSPENDED READY MESSAGE
flux-system main/5160fe3801b485575424d53616dff76fcc74e478 False False failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
gilogging main/adb4d09 False True stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'
❯ flux logs --level=error
❯ kubectl describe gitrepository flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: reconcile.fluxcd.io/requestedAt: 2022-08-17T10:52:38.057356-04:00
API Version: source.toolkit.fluxcd.io/v1beta2
Kind: GitRepository
Metadata:
Creation Timestamp: 2022-03-30T18:50:51Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Resource Version: 90866210
UID: 6e766830-bebd-4f26-a1e2-07232736d405
Spec:
Git Implementation: go-git
Interval: 1m0s
Ref:
Branch: main
Secret Ref:
Name: flux-system
Timeout: 20s
URL: ssh://[email protected]/gitops/k8s-configs
Status:
Artifact:
Checksum: 0bf60b79888c89ee8ba48d42a296f8fafb1a27fb8b5b8d319e253c1fc981299e
Last Update Time: 2022-08-15T18:49:12Z
Path: gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
Revision: main/5160fe3801b485575424d53616dff76fcc74e478
Size: 37517
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
Conditions:
Last Transition Time: 2022-08-16T16:40:35Z
Message: failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
Observed Generation: 1
Reason: GitOperationFailed
Status: False
Type: Ready
Last Transition Time: 2022-08-15T16:56:08Z
Message: stored artifact for revision 'main/5160fe3801b485575424d53616dff76fcc74e478'
Observed Generation: 1
Reason: Succeeded
Status: True
Type: ArtifactInStorage
Last Transition Time: 2022-08-16T16:40:35Z
Message: failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
Observed Generation: 1
Reason: GitOperationFailed
Status: True
Type: FetchFailed
Content Config Checksum: sha256:fcbcf165908dd18a9e49f7ff27810176db8e9f63b4352213741664245224f8aa
Last Handled Reconcile At: 2022-08-17T10:52:38.057356-04:00
Observed Generation: 1
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning GitOperationFailed 12m (x106 over 23h) source-controller failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue
@cdenneen Can you share more details about your GitLab instance? I'm running GitLab 15.2.2-ee and it's working fine.
I did these steps, using AWS EKS and CodeCommit, from the scratch:
eksctl create cluster --name flux-demo --nodegroup-name ng-flux
git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo
cd flux-demo
mkdir -p ./clusters/flux-demo/flux-system
flux install \
--components-extra=image-reflector-controller,image-automation-controller \
--export > ./clusters/flux-demo/flux-system/gotk-components.yaml
git add -A && git commit -m "add components" && git push
kubectl apply -f ./clusters/flux-demo/flux-system/gotk-components.yaml
flux check
aws iam create-user --user-name fluxserviceaccount
aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AWSCodeCommitFullAccess --user-name fluxserviceaccount
PUBLIC_KEY=$(cat ~/.ssh/flux_rsa.pub)
aws iam upload-ssh-public-key --user-name fluxserviceaccount --ssh-public-key-body $PUBLIC_KEY
flux create source git flux-system \
--git-implementation=libgit2 \
--url=ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo \
--branch=master \
--private-key-file=flux_rsa \
--interval=1m
and I got this error:
✗ failed to checkout and determine revision: unable to clone 'ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I am using:
- flux version 0.32.0
- source-controller:v0.26.1
- eksctl 0.109.0-dev+78243b4c.2022-08-19T12:49:29Z
- kubernetes 1.22
@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using flux create secret git
(ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use the flux create source --secret-ref
instead of --private-key-file
to authenticate.
@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/
@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using
flux create secret git
(ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use theflux create source --secret-ref
instead of--private-key-file
to authenticate.
@aryan9600 No, my key is not protected with a password, there is no passphrase. I was using private key because of the CodeCommit of AWS. I am going to try flux create secret git
. Thank you.
@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/
@stefanprodan Yes. The tutorial works with the HTTPS:
flux create source git flux-system \
--git-implementation=libgit2 \
--url=https://git-codecommit.<region>.amazonaws.com/v1/repos/<my-repository> \
...
but not with SSH, because o the issue:
flux create source git flux-system \
--git-implementation=libgit2 \
--url=ssh://Your-SSH-Key-ID@git-codecommit.<region>.amazonaws.com/v1/repos/<my-repository> \
...
thanks.