e2e - Verify cluster reconciliation failed - invalid 'ssh' auth option: 'identity' is required

[Ortub]

Hey, When the e2e is running I'm getting this error: flux-system gitrepository/flux-system False invalid 'ssh' auth option: 'identity' is required

what might be the reason for this failure?

[aidmax]

+1 using kind cluster and basic authentication to private GitHub repository

[kingdonb]

@aidmax I saw you posted about setting gitImplementation to libgit2 but retracted it – did you retract that because it didn't work, or another reason?

(I've seen a few people mentioning issues with private GitHub repositories, and I have thought to suggest testing the other Git implementation, but discounted the idea because I didn't think it would help... so I'm fishing to find out for this reason.)

[aidmax]

@aidmax I saw you posted about setting gitImplementation to libgit2 but retracted it – did you retract that because it didn't work, or another reason?

(I've seen a few people mentioning issues with private GitHub repositories, and I have thought to suggest testing the other Git implementation, but discounted the idea because I didn't think it would help... so I'm fishing to find out for this reason.)

Yes @kingdonb, it's working weirdly: first time I create git source everything is OK, I can even reconcile it repeatedly, but after I add first kustomization, the source returns this 'ssh' auth option: 'identity' is required error regardless of git-implementation.

See output below:

$ flux create source git flux-system \
--url=https://github.com/aidmax/fleet-infra.git \
--branch=main \
--username=$GITHUB_USER \
--password=$GITHUB_TOKEN \
--git-implementation=libgit2
✚ generating GitRepository source
► applying secret with repository credentials
✔ authentication configured
► applying GitRepository source
✔ GitRepository source created
◎ waiting for GitRepository source reconciliation
✔ GitRepository source reconciliation completed
✔ fetched revision: main/5c581488a44d073a1a0d7dfb8ec9f846864770aa

$ flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main/5c581488a44d073a1a0d7dfb8ec9f846864770aa

$ flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main/5c581488a44d073a1a0d7dfb8ec9f846864770aa
$
$ flux get ks
✗ no Kustomization objects found in flux-system namespace

$ flux create kustomization flux-system \
--source=flux-system \
--path=./clusters/pi-cluster
✚ generating Kustomization
► applying Kustomization
✔ Kustomization created
◎ waiting for Kustomization reconciliation
✔ Kustomization flux-system is ready
✔ applied revision main/5c581488a44d073a1a0d7dfb8ec9f846864770aa

$ flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to configure auth strategy for Git implementation 'libgit2': invalid 'ssh' auth option: 'identity' is required'

flux --version
flux version 0.28.2

kind --version
kind version 0.11.1

[devantler]

Any updates to this? Did you figure out how to circumvent the issue?

[darioblanco]

I think the problem resides in the gitrepository resource that flux-system creates. If you go to gotok-sync.yaml, you will see a resource like this

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: ssh://[email protected]/darioblanco/gitops

Where the url is the link to your flux2-kustomize-helm-example fork. But when using flux create source git, we provide a username and password, which basically requires to pass the repository as https and not as ssh.

Based on the documentation:

The create source git command generates a GitRepository resource and waits for it to sync. For Git over SSH, host and SSH keys are automatically generated and stored in a Kubernetes secret. For private Git repositories, the basic authentication credentials are stored in a Kubernetes secret.

I assume that the example does not provide the username and password because it generates the repo with the SSH protocol and it just requires the permissions to add the keys to that repo (so it can connect). Nonetheless, if you add a different source that is not SSH (or that does not allow SSH access), you need to provide those secrets in some way and that's why I see examples where username and password is provided (because they have apps that are configured via HTTPS). That collides with this command.

I think that one solution would be to keep this command without basic auth (so that it connects to flux-system via SSH) and to create a different system scoped per repo in case that somebody wants to pull private repos with basic auth.

For instance, removing the basic auth from the original command flux create source git flux-system and adding this might work:

          flux create secret git podinfo-auth \
            --url=https://github.com/stefanprodan/podinfo \
            --username=${GITHUB_USER} \
            --password=${GITHUB_TOKEN}

If you run without the username and password credentials, you get this error:

Warning AuthenticationFailed 101s (x9 over 4m53s) source-controller failed to get secret 'flux-system/flux-system': secrets "flux-system" not found

So you need to configure the credentials to be able to access the required repos via SSH (or change the way in which that resource is generated)

EDIT: I fixed it just removing my automatically generated files from ./clusters/**/flux-system/, for some reason the --ignore-paths="./clusters/**/flux-system/" does not ignore that folder and it takes the resources from it. It works in the example repo because that folder is empty.

[devantler]

Thanks, your last comment about it not ignoring the generated files makes so much sense. Much appreciated from my end! Looking forward to the fix being merged. Great work!

[wilsonchiu]

Any updates on this? I have forked a private repo on GitHub to follow the examples, and I would really like to get the e2e steps working with GitHub Actions.

My gotk-sync.yaml specifies the url as url: ssh://[email protected]/wilsonchiu/flux2-kustomize-helm-test, so I have tried to modify e2e.yaml to use ssh, such as --url=ssh://[email protected]/${{ github.event.repository.full_name}}. However this just causes a different issue, where flux create source git generates a different public key for every run, I can't figure out how to use add this as Deploy Key on GitHub.

I have tried a few other tweaks such as adding --silent, wondering if flux would use the Deploy Key already on GitHub from a local bootstrap, but this doesn't work.

I have also tried creating a secret locally and adding the public key as Deploy Key on GitHub, again this doesn't work but with a different response. If I do something like flux create secret git repo-secret --url:ssh://[email protected]/wilsonchiu/flux2-kustomize-helm-test and grab the public key. When I run e2e workflow using --secret-ref repo-secret flux tells me failed to get secret 'flux-system/repo-secret': secrets "repo-secret" not found.

I am probably missing something straight forward regarding accessing private GitHub repos using flux from GitHub Actions, but I have worked on this for a few days now, and I cannot find a clear example on how to do this. I would be most grateful for any suggestions.

[stefanprodan]

This has been fixed in https://github.com/fluxcd/flux2-kustomize-helm-example/pull/99 please update your e2e workflow.

[wilsonchiu]

This has been fixed in #99 please update your e2e workflow.

Many thanks @stefanprodan for your comments. To wind back a little and to create a MRE, I just cloned the repo again, once as a private repo and again as a public repo. The e2e and test workflows were triggered and they ran to completion successfully on the public repo, but the private repo fails the e2e workflow at the waiting for GitRepository source reconciliation step with:

failed to checkout and determine revision: unable to clone 'https://github.com/wilsonchiu/flux2-test': authentication required

I am struggling to figure out from the documentation how to use flux create source git in this scenario - how to specify authentication methods for flux, and how to get the key/token to flux. If I understand correctly, flux is being invoked by GitHub Actions which is on a GitHub hosted instance and it should have access to the repo. Basically, what do I need to tell 'flux create source git`?

[stefanprodan]

Add --password=${{ secrets.GITHUB_TOKEN }} if the repo is private.

[wilsonchiu]

Thanks again @stefanprodan , this is what I now have in the e2e.yaml but I still can't get past the flux create source git step.

          flux create source git flux-system \
          --url=${{ github.event.repository.html_url }} \
          --password=${{ secrets.GITHUB_TOKEN }} \
          --branch=${GITHUB_REF#refs/heads/} \
          --ignore-paths="clusters/**/flux-system/"

Error in raw log from GitHub Actions run:

2023-08-18T09:46:06.6976237Z ##[group]Run flux create source git flux-system \
2023-08-18T09:46:06.6976661Z [36;1mflux create source git flux-system \[0m
2023-08-18T09:46:06.6977057Z [36;1m--url=https://github.com/wilsonchiu/flux2-test \[0m
2023-08-18T09:46:06.6978153Z [36;1m--*** \[0m
2023-08-18T09:46:06.6978454Z [36;1m--branch=${GITHUB_REF#refs/heads/} \[0m
2023-08-18T09:46:06.6978821Z [36;1m--ignore-paths="clusters/**/flux-system/"[0m
2023-08-18T09:46:06.6979185Z [36;1mflux create kustomization flux-system \[0m
2023-08-18T09:46:06.6979525Z [36;1m--source=flux-system \[0m
2023-08-18T09:46:06.6979834Z [36;1m--path=./clusters/staging[0m
2023-08-18T09:46:06.7042164Z shell: /usr/bin/bash -e {0}
2023-08-18T09:46:06.7042485Z ##[endgroup]
2023-08-18T09:46:06.7413877Z ✚ generating GitRepository source
2023-08-18T09:46:06.7415548Z ► applying GitRepository source
2023-08-18T09:46:06.7601054Z ✔ GitRepository source created
2023-08-18T09:46:06.7601846Z ◎ waiting for GitRepository source reconciliation
2023-08-18T09:46:08.7757725Z ✗ failed to checkout and determine revision: unable to clone 'https://github.com/wilsonchiu/flux2-test': authentication required
2023-08-18T09:46:08.7787468Z ##[error]Process completed with exit code 1.

[stefanprodan]

Ok can you please add --username=${GITHUB_ACTOR} also make sure the token has read access to the repo, by setting this in the job:

    permissions:
      contents: read

[wilsonchiu]

Success! The following snippet in e2e.yaml works for a private GitHub repo. I did not add the permissions block to the yaml, but I did check from the GitHub Web UI that the repo Actions settings has Read repository contents and packages permissions checked under Workflow permissions, which seems to be the default.

          flux create source git flux-system \
          --url=${{ github.event.repository.html_url }} \
          --username=${GITHUB_ACTOR} \
          --password=${{ secrets.GITHUB_TOKEN }} \
          --branch=${GITHUB_REF#refs/heads/} \
          --ignore-paths="clusters/**/flux-system/"

Thank you @stefanprodan again for the assistance and your patience in helping me to gain further understanding of flux.