Using git credentials in sidecar plugin

[patoarvizu]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I have a sidecar plugin for Tanka, that uses jsonnet-bundler (jb) to pull dependencies from private repos. I have the ConfigManagementPlugin manifest configured to have init run jb install, which is how jsonnet-bundler pulls its dependencies, and it calls git under the covers. I assume that because these calls to git are unknown to/unmanaged by Argo, the pre-configured credentials aren't available, but up until 2.2.5, I was able to have a workaround this issue by mounting a git-ask-pass.sh script via a ConfigMap, and setting GIT_USERNAME and GIT_PASSWORD as environment variables on the sidecar container. Once I tried to upgrade to 2.3.1, this stopped working. (Note: Argo was still able to clone the private repo that my Application was pointing to, but what stopped working was the jb update command that runs in the sidecar as part of the initialization.)

I see here that argocd-git-ask-pass was introduced, but I don't quite understand if this is something that should be available in custom sidecars, or how to use it. It would be good if I can understand why the upgrade to 2.3.1 broke my workaround, but I'd be satisfied with a path forward if that's all we can find.

To Reproduce

I'm installing via the Helm chart, here are the relevant values:

repoServer:
  volumes:
  - configMap:
      name: tanka-plugin
    name: tanka-plugin
  - configMap:
      name: git-creds-config
      items:
      - key: git-ask-pass.sh
        path: git-ask-pass.sh
        mode: 0755
    name: git-creds-config
  extraContainers:
  - name: tanka
    command: [/var/run/argocd/argocd-cmp-server]
    image: grafana/tanka:0.19.0
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
      runAsUser: 999
    volumeMounts:
    - mountPath: /var/run/argocd
      name: var-files
    - mountPath: /home/argocd/cmp-server/plugins
      name: plugins
    - mountPath: /home/argocd/cmp-server/config/plugin.yaml
      subPath: tanka-plugin.yaml
      name: tanka-plugin
    - mountPath: /tmp
      name: tmp-dir
    - mountPath: /usr/local/bin/git-ask-pass.sh
      subPath: git-ask-pass.sh
      name: git-creds-config
    env:
    - name: GIT_USERNAME
      valueFrom:
        secretKeyRef:
          name: argocd-repo-creds-ghe
          key: username
    - name: GIT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: argocd-repo-creds-ghe
          key: password

The corresponding ConfigMaps are as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tanka-plugin
  namespace: argocd
data:
  tanka-plugin.yaml: |-
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: tanka-plugin
    spec:
      version: v1.0
      init:
        command: [sh, -c, 'jb install']
      generate:
        command: [sh, -c, 'tk show ${TK_ENV} --dangerous-allow-redirect ${EXTRA_ARGS}']
      discover:
        fileName: jsonnetfile.json
      allowConcurrency: true
      lockRepo: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: git-creds-config
  namespace: argocd
data:
  git-ask-pass.sh: |-
    #!/bin/sh
    case "$1" in
    Username*) echo "${GIT_USERNAME}" ;;
    Password*) echo "${GIT_PASSWORD}" ;;
    esac

Expected behavior

1. I would expect that the workaround above still works in 2.3.1 as it did in 2.2.5, leveraging git's GIT_ASKPASS functionality.
2. If this new argocd-git-ask-pass is available for custom sidecar plugins, I'd like to understand how to use it.

Screenshots

Version

I'm not running the argocd command line, but I'm using Argo 2.3.1

Logs

[patoarvizu]

Some additional information: I had a hunch and tried using a custom image (instead of the upstream grafana/tanka) that copied /usr/local/bin/argocd from quay.io/argoproj/argocd:v2.3.1. When I did this, I got the following error message:

... level=fatal msg="rpc error: code = NotFound desc = unknown nonce" error: unable to read askpass response from 'argocd' fatal: could not read Username...

Which sounds like there's some misconfiguration somewhere, I just don't know where.

One thing I do want to point out though is that if this is the way that sidecars have to be configured to be able to get git credentials, then I feel like this cancels out the benefit of using sidecar plugins, which is not having to create custom images. For example, in this case I wouldn't be able to use the upstream grafana/tanka image on a sidecar cleanly, and would have to create my own image that includes argocd. Or am I missing something?

[dayyeung]

also having this issue with classic cmp after upgrading to v2.3 before v2.3 git clone (https) works without any special code seems to be the same cause for #7995 #8389 seems to be a valid fix

[kayleighmcginley]

I'm also seeing this issue after updating to v2.3+, any apps using our tanka plugin fail with the same error message reported by @patoarvizu. Watching for updates on https://github.com/argoproj/argo-cd/pull/8389. Ty.

[patoarvizu]

@crenshaw-dev Is #8389 the issue to track now for the resolution to this?

[crenshaw-dev]

@patoarvizu that's the PR to track (just merged), https://github.com/argoproj/argo-cd/issues/7995 is the issue.

I'm going to put up a PR to cherry-pick this onto the next 2.3 release.

[patoarvizu]

Awesome, thank you! 👍

[crenshaw-dev]

@patoarvizu have you tested this for your sidecar plugin? After reviewing the code with @leoluz I'm not convinced this will work for sidecars.

[patoarvizu]

@crenshaw-dev No, I haven't tested it. I was waiting for 2.3.4 to be formally released to give it a try.

[crenshaw-dev]

Okay cool. Will test. We might want to make the longer-lived credentials opt-in, because it's a bit less secure, and many folks won't need git credentials during manifest generation. Will keep this thread up to date.

[crenshaw-dev]

Reopening because I think this needs some more thought/work on our side.

1. I'd rather not implement the fix for argocd-cm (i.e. non-sidecar plugins). We're deprecating argocd-cm plugins in 2.4 and strongly encouraging folks to move to sidecars due to much better security.
2. I want to look more carefully at when, why, and how we communicate git creds to CMPs. Specifically I'd like to avoid communicating creds at all by default. Many plugins won't use git, and sending the creds only risks them being stolen. Making cred passing opt-in (by the CMP author) will ensure the author thinks about how to secure them.

[leoluz]

I wanted to clarify @crenshaw-dev comment above a bit more. There is recent security related work done in CMP sidecar that detaches its file system from the one used by repo-server. This change is currently in master and will be available in the next release. That being said, if there is a specific file required by a plugin, it needs to be available in the Application's source repo. Credentials will be passed as environment variables available in the process that executes the plugin's command. We have a e2e test to validate private repo access in CMP sidecar and it passes. Maybe jb update isn't able to get the necessary credentials currently available. This needs further investigation.

@patoarvizu can you please provide the full Application manifest that you are applying to reproduce the issue? Is this easily reproducible locally?

[patoarvizu]

I can create a reproduction repo for you, including the full Application manifest, but it may need to be forked/edited to make it private to validate the credentials work as expected. I should be able to create it for you in the next day or two.

[patoarvizu]

@leoluz @crenshaw-dev As promised, here's the reproduction repo: https://github.com/patoarvizu/argo-sidecar-plugin-creds-repro

It may need some tweaks to get it to work since it depends on access to private repos, but I think you should be able to take it from here. I added instructions for how to reproduce locally using a k3d cluster if it makes it easier, but I believe the Helm setup + the sample Application manifest there should be enough for you.

Please let me know if there's anything else I can help you with!

[crenshaw-dev]

Wow this is brilliant. Thanks, @patoarvizu! Will give it a try Monday.

[RixTmobilender]

Also a tanka user and just got bit by this, since argo overwrites GIT_ASKPASS in here. I worked around it by manually exporting GIT_ASKPASS to my mounted helper within the init command. E.g:

export GIT_ASKPASS=/usr/local/bin/git-ask-pass.sh &&
jb install

[notfromstatefarm]

Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.

On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.

[patoarvizu]

For what it's worth, I was able to upgrade to 2.2.8 without issues.

[HariSekhon]

I've used Git's AskPass to inject credentials in a similar case.

Git askpass script in a configmap:

https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml

patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:

https://github.com/HariSekhon/Kubernetes-configs/blob/master/argocd/base/argocd-git-askpass.repo-server.jsonpatch.yaml

[leoluz]

@HariSekhon can you please confirm the ArgoCD version you are running? Thank you.

[HariSekhon]

@leoluz I did this on an ArgoCD 2.0 that I use in production to solve my Kustomize external bases not working.

[zephyros-dev]

I've used Git's AskPass to inject credentials in a similar case.

Git askpass script in a configmap:

https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml

patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:

https://github.com/HariSekhon/Kubernetes-configs/blob/master/argocd-git-askpass.repo-server.jsonpatch.yaml

I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).

[HariSekhon]

I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).

In the above simple example this is true, but actually the git askpass mechanism can pass in variables, which means you can add logic to the script to return different credentials based on different protocol, host and repo path:

https://git-scm.com/docs/git-credential#IOFMT

[jannfis]

Reading through this issue, I think this bug affects only plugins that use kind of git integration themselves, right? E.g. those which need access to Git after Argo CD has provided the set of manifests it rendered to the plugin.

[crenshaw-dev]

@jannfis that is my understanding as well. I do not believe that plugins which do not use git (either directly or indirectly) are impacted.

[kxs-sindrakumar]

Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.

On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.

same with me. Works on 2.2.5 fine. @crenshaw-dev any chance we can get a fix in for the latest version?

[cyrus-mc]

This is an issue still and is blocking me from upgrading past 2.2.x version.

In 2.2 git-ask-pass.sh were trivial as the GIT_USERNAME and GIT_PASSWORD were exposed via the environment.

In 2.5 that isn't the case.

[abkura]

This is blocking our organization from being on the latest version and inturn not able to use new features (Argocd notifications)

[crenshaw-dev]

This is definitely still top of my mind. I'll do what I can to get something ready before 2.6. The solution has to prioritize security and Argo CD admin control over credentials. I want to take time to get that right.

[abkura]

@crenshaw-dev, do you know if there's any progress on the above issue?

[quentinleclerc]

I've used Git's AskPass to inject credentials in a similar case. Git askpass script in a configmap: https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want: https://github.com/HariSekhon/Kubernetes-configs/blob/master/argocd-git-askpass.repo-server.jsonpatch.yaml

I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).

Hello @zephyros-dev,

Could you please share what you've done for sidecar please? I'm facing the same issue and just can't get it working. Having it work for one single credentials would be enough for me waiting for the resolution (which does not seem to be coming pretty soon) Or if someone else from here has a workaround!

Thanks!