argocd-vault-plugin Unable to get argocd-vault-plugin working with SOPS

Unable to get argocd-vault-plugin working with SOPS

Open jonathon2nd opened this issue 2 years ago • 4 comments

Describe the bug I am struggling to get argocd-vault-plugin working with sops for my install of argocd. I am sure it is an error on my part, but have not been able to figure it out and hoping someone can point me in the right direction.

To Reproduce Install configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
  namespace: argocd
data:
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ anything is done, therefore the logic is to check
      # if this looks like a Kustomize bundle
      discover:
        find:
          command:
            - find
            - "."
            - -name
            - kustomization.yaml
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . | argocd-vault-plugin generate -"
      lockRepo: false
  avp-helm.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ any Helm templating is done, therefore the logic is to check
      # if this looks like a Helm chart
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        # **IMPORTANT**: passing `${ARGOCD_ENV_HELM_ARGS}` effectively allows users to run arbitrary code in the Argo CD 
        # repo-server (or, if using a sidecar, in the plugin sidecar). Only use this when the users are completely trusted. If
        # possible, determine which Helm arguments are needed by your users and explicitly pass only those arguments.
        command:
          - sh
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |
            argocd-vault-plugin generate -
      lockRepo: false
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false

Install sop secret (this was done during my debugging)

Patch argo-cd

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:

      # Mount SA token for Kubernets auth
      # Note: In 2.4.0 onward, there is a dedicated SA for repo-server (not default)
      # Note: This is not fully supported for Kubernetes < v1.19
      automountServiceAccountToken: true

      # Each of the embedded YAMLs inside cmp-plugin ConfigMap will be mounted into it's respective plugin sidecar
      volumes:
        - configMap:
            name: cmp-plugin
          name: cmp-plugin
        - name: custom-tools
          emptyDir: {}
        - name: sops-age
          secret:
            secretName: sops-age

      # Download tools
      initContainers:
      - name: download-tools
        image: registry.access.redhat.com/ubi8
        env:
          - name: AVP_VERSION
            value: 1.16.1
        command: [sh, -c]
        args:
          - >-
            curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/

        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
            
      # argocd-vault-plugin with Helm
      containers:
      - name: avp-helm
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.8.4
        env:
        - name: AVP_TYPE
          value: "sops"
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp-helm.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

          - mountPath: /.config/sops/age/keys.txt
            name: sops-age
            subPath: age-key.txt


      # argocd-vault-plugin with Kustomize
      - name: avp-kustomize
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.8.4
        env:
        - name: AVP_TYPE
          value: "sops"
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp-kustomize.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin
          
          - mountPath: /.config/sops/age/keys.txt
            name: sops-age
            subPath: age-key.txt

      # argocd-vault-plugin with plain YAML
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.8.4
        env:
        - name: AVP_TYPE
          value: "sops"
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp.yaml
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

          - mountPath: /.config/sops/age/keys.txt
            name: sops-age
            subPath: age-key.txt

test secret

apiVersion: v1
kind: Secret
metadata:
    name: test-secret
    annotations:
        avp.kubernetes.io/path: "secret-test.enc.yaml"
stringData:
    PASSWORD: sfnasomh8e9cyt7a248yth7uy4w

sops --encrypt templates/secret-test.yaml > templates/secret-test.enc.yaml

Then add the following to the chart for test install along with the output of the sops command.

---
apiVersion: v1
kind: Secret
metadata:
    name: test-secret
    annotations:
        avp.kubernetes.io/path: "secret-test.enc.yaml"
stringData:
    PASSWORD: <PASSWORD>

Expected behavior I expect to see the decrypted value in the secret in k8s.

Screenshots/Verbose output

No errors in AVP-helm


Container: avp-helm
Filter
Connected
time="2023-11-01T18:57:53Z" level=info msg="ArgoCD ConfigManagementPlugin Server is starting" built="2023-09-13T19:12:09Z" commit=c27929928104dc37b937764baf65f38b78930e59 version=v2.8.4+c279299
time="2023-11-01T18:57:53Z" level=info msg="argocd-cmp-server v2.8.4+c279299 serving on /home/argocd/cmp-server/plugins/argocd-vault-plugin-helm.sock"
time="2023-11-01T19:07:53Z" level=info msg="Alloc=8016 TotalAlloc=16670 Sys=33661 NumGC=7 Goroutines=9"
time="2023-11-01T19:17:53Z" level=info msg="Alloc=8016 TotalAlloc=16683 Sys=33661 NumGC=12 Goroutines=9"
time="2023-11-01T19:27:53Z" level=info msg="Alloc=8016 TotalAlloc=16697 Sys=33661 NumGC=17 Goroutines=9"
time="2023-11-01T19:37:53Z" level=info msg="Alloc=8017 TotalAlloc=16710 Sys=33661 NumGC=22 Goroutines=9"
time="2023-11-01T19:47:53Z" level=info msg="Alloc=8017 TotalAlloc=16724 Sys=33661 NumGC=27 Goroutines=9"
time="2023-11-01T19:57:53Z" level=info msg="Alloc=8017 TotalAlloc=16737 Sys=33661 NumGC=31 Goroutines=9"
time="2023-11-01T20:07:53Z" level=info msg="Alloc=8017 TotalAlloc=16751 Sys=33661 NumGC=36 Goroutines=9"
time="2023-11-01T20:17:53Z" level=info msg="Alloc=8017 TotalAlloc=16764 Sys=33661 NumGC=41 Goroutines=9"
time="2023-11-01T20:21:03Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/fd1289c8-51f3-4961-82ef-86f19ba7448a/charts/nodepool-annotater execID=dabe4
time="2023-11-01T20:21:03Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:21:03Z" grpc.time_ms=367.146 span.kind=server system=grpc
time="2023-11-01T20:21:03Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/15b8c279-8881-4a07-8fc3-c40f90b9b75b/charts/nodepool-annotater execID=4980a
time="2023-11-01T20:21:03Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:21:03Z" grpc.time_ms=324.603 span.kind=server system=grpc
time="2023-11-01T20:21:04Z" level=info msg="Generating manifests with no request-level timeout"
time="2023-11-01T20:21:04Z" level=info msg="sh -c \"helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\\nargocd-vault-plugin generate -\\n\"" dir=/tmp/_cmp_server/c71ca19e-cf54-46c8-89a7-93276d7f2ac4/charts/nodepool-annotater execID=7f785
time="2023-11-01T20:21:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:21:03Z" grpc.time_ms=395.429 span.kind=server system=grpc
time="2023-11-01T20:24:03Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/98c3cf09-aa70-462d-abc7-fff4bd81408d/charts/nodepool-annotater execID=44c57
time="2023-11-01T20:24:03Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:24:03Z" grpc.time_ms=349.544 span.kind=server system=grpc
time="2023-11-01T20:24:04Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/6f30d5fb-6d09-424c-b56b-b101826b0e8d/charts/nodepool-annotater execID=94004
time="2023-11-01T20:24:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:24:03Z" grpc.time_ms=308.315 span.kind=server system=grpc
time="2023-11-01T20:24:04Z" level=info msg="Generating manifests with no request-level timeout"
time="2023-11-01T20:24:04Z" level=info msg="sh -c \"helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\\nargocd-vault-plugin generate -\\n\"" dir=/tmp/_cmp_server/d67ccc30-fb02-4568-98a2-b7a55e28cf88/charts/nodepool-annotater execID=f61c8
time="2023-11-01T20:24:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:24:04Z" grpc.time_ms=432.205 span.kind=server system=grpc
time="2023-11-01T20:27:04Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/382e691b-8ac6-43fd-b689-0c6d3897f7bc/charts/nodepool-annotater execID=40df6
time="2023-11-01T20:27:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:27:04Z" grpc.time_ms=301.414 span.kind=server system=grpc
time="2023-11-01T20:27:04Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/c764f330-c012-4815-93b4-c285716cbf45/charts/nodepool-annotater execID=e9a44
time="2023-11-01T20:27:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:27:04Z" grpc.time_ms=306.126 span.kind=server system=grpc
time="2023-11-01T20:27:05Z" level=info msg="Generating manifests with no request-level timeout"
time="2023-11-01T20:27:05Z" level=info msg="sh -c \"helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\\nargocd-vault-plugin generate -\\n\"" dir=/tmp/_cmp_server/e423a382-6049-4e12-b0d5-dccb0cd96024/charts/nodepool-annotater execID=abaed
time="2023-11-01T20:27:05Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T20:27:04Z" grpc.time_ms=422.821 span.kind=server system=grpc
time="2023-11-01T20:27:53Z" level=info msg="Alloc=14475 TotalAlloc=325876 Sys=46717 NumGC=69 Goroutines=9"
time="2023-11-01T20:37:53Z" level=info msg="Alloc=8131 TotalAlloc=325877 Sys=46717 NumGC=74 Goroutines=9"
time="2023-11-01T20:47:53Z" level=info msg="Alloc=8131 TotalAlloc=325890 Sys=46717 NumGC=79 Goroutines=9"
time="2023-11-01T20:57:53Z" level=info msg="Alloc=8131 TotalAlloc=325904 Sys=46717 NumGC=83 Goroutines=9"
time="2023-11-01T21:06:03Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/364f4042-9e2a-4056-853f-88c1cd69ef6c/charts/nodepool-annotater execID=ad513
time="2023-11-01T21:06:03Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:03Z" grpc.time_ms=344.671 span.kind=server system=grpc
time="2023-11-01T21:06:04Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/8924bd9e-1347-41a6-9477-f2e9c941ce07/charts/nodepool-annotater execID=1ee5d
time="2023-11-01T21:06:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:03Z" grpc.time_ms=287.75 span.kind=server system=grpc
time="2023-11-01T21:06:04Z" level=info msg="finished streaming call with code Canceled" error="generate manifest error receiving stream: error receiving stream header: rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:04Z" grpc.time_ms=164.602 span.kind=server system=grpc
time="2023-11-01T21:06:04Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/35589cde-2c66-4e74-adfb-b562388c8352/charts/nodepool-annotater execID=46c2a
time="2023-11-01T21:06:04Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:04Z" grpc.time_ms=367.573 span.kind=server system=grpc
time="2023-11-01T21:06:05Z" level=info msg="sh -c \"find . -name 'Chart.yaml' && find . -name 'values.yaml'\"" dir=/tmp/_cmp_server/ce218d02-0dd0-4906-951d-0b1e4b3f078b/charts/nodepool-annotater execID=e26c8
time="2023-11-01T21:06:05Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:04Z" grpc.time_ms=309.409 span.kind=server system=grpc
time="2023-11-01T21:06:05Z" level=info msg="Generating manifests with no request-level timeout"
time="2023-11-01T21:06:05Z" level=info msg="sh -c \"helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\\nargocd-vault-plugin generate -\\n\"" dir=/tmp/_cmp_server/e897bad9-e3d8-464f-b858-4853ee57e58d/charts/nodepool-annotater execID=68b0b
time="2023-11-01T21:06:05Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2023-11-01T21:06:05Z" grpc.time_ms=435.699 span.kind=server system=grpc
time="2023-11-01T21:07:53Z" level=info msg="Alloc=12043 TotalAlloc=503623 Sys=46717 NumGC=101 Goroutines=9"

Additional context

Nov 01 '23 21:11 jonathon2nd

I am wondering if SOPS is not fully supported yet, as it is not listed here: https://argocd-vault-plugin.readthedocs.io/en/stable/config/#full-list-of-supported-parameters

Nov 01 '23 21:11 jonathon2nd

argocd-vault-plugin argocd-vault-plugin copied to clipboard

Unable to get argocd-vault-plugin working with SOPS

argocd-vault-plugin
argocd-vault-plugin copied to clipboard