Unable to use Github App repo-cred for git update method

[timothyjlaurent]

Describe the bug I'm running into some issues with ArgoCD image updater segfaulting and being unable to write back to the repository.

To Reproduce I'm using a GitHub app for the repo credentials:

apiVersion: v1
data:
  githubAppID: ...
  githubAppInstallationID: ...
  githubAppPrivateKey: LS...
  url: ...
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repo-creds
  name: argocd-github-app-secret
  namespace: argocd
type: Opaque

This is working for argocd to pull manifests from our internal repos, the GitHub app has also been given write permissions to these repos.

When the image updater runs it has an error and enters into a crash loop.

Expected behavior I expect it to write to the repo without crashing using the GitHub App repo creds Additional context Add any other context about the problem here.

Version Argocd-image-updater v0.12.2 Argocd 2.7.4

Logs Please paste any relevant logs here

time="2023-08-23T01:57:43Z" level=info msg="Committing 1 parameter update(s) for application my-app-dev" application=my-app-dev
time="2023-08-23T01:57:43Z" level=info msg="Starting configmap/secret informers"
time="2023-08-23T01:57:43Z" level=info msg="Configmap/secret informer synced"
time="2023-08-23T01:57:43Z" level=info msg="Initializing https://github.com/my-org/gitops to /tmp/git-my-app-dev415308282"
time="2023-08-23T01:57:43Z" level=info msg="rm -rf /tmp/git-my-app-dev415308282" dir= execID=cd83d
time="2023-08-23T01:57:43Z" level=info msg="configmap informer cancelled"
time="2023-08-23T01:57:43Z" level=info msg="secrets informer cancelled"
time="2023-08-23T01:57:43Z" level=info msg=Trace args="[rm -rf /tmp/git-my-app-dev415308282]" dir= operation_name="exec rm" time_ms=0.8027190000000001
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xc4e6fc]

goroutine 209 [running]:
github.com/argoproj/argo-cd/v2/util/git.GitHubAppCreds.Environ({0x4ca29, 0x25c7f60, {0xc000a5ce00, 0x68b}, {0x0, 0x0}, {0xc000946520, 0x20}, {0x0, 0x0},
    /go/pkg/mod/github.com/argoproj/argo-cd/[email protected]/util/git/creds.go:388 +0xa5c
github.com/argoproj-labs/argocd-image-updater/ext/git.(*nativeGitClient).runCredentialedCmd(0xc000790150, {0x24b1ddb?, 0xc0009a56b0?}, {0xc00054b2d8?, 0
    /src/argocd-image-updater/ext/git/client.go:595 +0x8b
...

This happens to be in the following line of the creds.go

	nonce := g.store.Add(githubAccessTokenUsername, token)

https://github.com/argoproj/argo-cd/blob/0ee33e52dd1f1bb944488584fc6f854b929f1180/util/git/creds.go#L388

[sockmister]

Instead of relying on the argocd-image-updater to re-use the credentials, I found that referencing the secret directly fixed this issue for me. ie.

before

metadata:
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git

after

metadata:
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/argocd-github-app-secret

Having said that, it sounds like a bug somewhere.

[djl2]

I'm having the same problem and errors now as well. Everything was working fine until I had to update my secret and restart the Image Updater. I changed nothing else.

Version Argocd-image-updater: latest ArgoCD v2.8.0+804d4b8

I implemented the workaround recommended by @sockmister. It cleared the crash loop and log errors, but it is still unable to write to the GitHub repo.

[timothyjlaurent]

For me, it turns out that we were using the 'latest' image updater image. Once we pinned to v0.12.2 it fixed the problem and we were able to reuse the credentials.

[djl2]

For me, it turns out that we were using the 'latest' image updater image. Once we pinned to v0.12.2 it fixed the problem and we were able to reuse the credentials.

Thanks a lot!! I was using the 'latest' image, too. To fix, I set the deployment in /manifest/install.yaml to pull the Release version, then redeployed it.

image: quay.io/argoprojlabs/argocd-image-updater:v0.12.2

The original crash loop and errors are gone now. Able to re-use the credentials and Image Updater works as expected.

Hopefully the bug in the latest build will be fixed soon. Just realized I had been using the latest build from May 2023, and some the newer features aren't available in the Release version from Jan 21, 2023.

[goern]

@timothyjlaurent how did the argocd-github-app-secret secret get created? is that part of an argocd deployment procedure I just havent see?

[timothyjlaurent]

@goern I created that secret myself per the instructions in ArgoCD's declarative setup docs.

[goern]

for reference, I am running Red Hat OpenShift GitOps 1.9.1 which is Argo CD 2.7.6

The Git Ops operator does not create any secret equivalent to https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#github-app-repositories

[BBQigniter]

could it be that this still happens with 0.13.0?