plugin installed via sidecar not working

[lixiaomei-git]

I installed argocd-vault-plugin via sidecar, and I could generated the secret directly through the avp container in argocd-repo-server pod, but the same secret can't be injected automaticlly when creating it by argocd application. Following is my steps:

1. I have already prepared the approle and secret in vault, and created the secret named vault-configuration to save the avp configurations.
2. Log into the avp container: k exec -it argocd-repo-server-cf6d987f4-d47hs -c avp -- sh In the container, I checked the plugin.yaml:

sh-4.4$ cat /home/argocd/cmp-server/config/plugin.yaml 
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: argocd-vault-plugin
spec:
  allowConcurrency: true
  discover:
    find:
      command:
        - sh
        - "-c"
        - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
  generate:
    command:
      - argocd-vault-plugin
      - generate
      - "-s"
      - "vault-configuration"
      - "."
  lockRepo: false

3. Created the secret.yaml:

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  annotations:
    avp.kubernetes.io/path: "kv/data/poc/argocd/example-secret"
type: Opaque
stringData:
  password: <example-secret>

4. Ran argocd-vault-plugin directly, and I got the correct password!

sh-4.4$ argocd-vault-plugin generate -s vault-configuration .
(0x2954920,0xc00080d2f0)apiVersion: v1
kind: Secret
metadata:
  annotations:
    avp.kubernetes.io/path: kv/data/poc/argocd/example-secret
  name: example-secret
stringData:
  password: test
type: Opaque
---

So I think the vault-configuration is correct.

However the plugin was not working when creating argocd application. Following is the application manifest:

kind: Application
metadata:
  name: test-app
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'xxxxxx'
    targetRevision: HEAD
    path: testapp

And in the testapp directory, I put the same secret.yaml. After applied, I checked the generated secret, the value is still . I don't know how to find the root cause.

Following is configmap for ConfigManagementPlugin:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmp-plugin
data:
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "-s"
          - "vault-configuration"
          - "."
      lockRepo: false

following is the argocd-repo-server patch file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      automountServiceAccountToken: true
      volumes:
        - configMap:
            name: argocd-cmp-plugin
          name: argocd-cmp-plugin
        - name: custom-tools
          emptyDir: { }
      initContainers:
        - name: download-tools
          image: registry.access.redhat.com/ubi8
          securityContext:
            allowPrivilegeEscalation: false
          env:
            - name: AVP_VERSION
              value: 1.11.0
          command: [ sh, -c ]
          args:
            - >-
              cd custom-tools &&
              curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
              chmod +x argocd-vault-plugin
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      containers:
        - name: avp
          command: [ /var/run/argocd/argocd-cmp-server ]
          image: registry.access.redhat.com/ubi8
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 999
          volumeMounts:
            - mountPath: /var/run/argocd
              name: var-files
            - mountPath: /home/argocd/cmp-server/plugins
              name: plugins
            - mountPath: /tmp
              name: tmp

            # Register plugins into sidecar
            - mountPath: /home/argocd/cmp-server/config/plugin.yaml
              subPath: avp.yaml
              name: argocd-cmp-plugin

            # Important: Mount tools into $PATH
            - name: custom-tools
              subPath: argocd-vault-plugin
              mountPath: /usr/local/bin/argocd-vault-plugin

[lixiaomei-git]

@jkayani Hi Josh, I think you are an expert of the plugin, could you please help me?

[BMonsalvatge]

Try updating your App manifest with plugin: {} like the following:

kind: Application
metadata:
  name: test-app
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'xxxxxx'
    targetRevision: HEAD
    path: testapp
    plugin: {}

Also are you getting a specific error message when trying to sync?

[lixiaomei-git]

Try updating your App manifest with plugin: {} like the following:

kind: Application
metadata:
  name: test-app
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'xxxxxx'
    targetRevision: HEAD
    path: testapp
    plugin: {}

Also are you getting a specific error message when trying to sync?

@BMonsalvatge Hi Bryan, thanks for your suggestion. I didn't get any error when trying to sync. I added plugin: {} just now and yes, things became different, the plugin ran. However I got an error. I was confused, I obversely checked the installation documents several times, it said that with sidecar installation, # No need to define plugin since Argo CD will figure it out!

[BMonsalvatge]

Mind sharing the new error?

[lixiaomei-git]

Mind sharing the new error?

couldn't find cmp-server plugin supporting repository But I have executed the find command in the testapp directory, I can got the expected result.

/testapp$ find . -name '*.yaml' | xargs -I {} grep "<path\\|avp\\.kubernetes\\.io" {} | grep .
    avp.kubernetes.io/path: "kv/data/poc/argocd/example-secret"

And yesterday, I installed the plugin using argocd-cm configmap instead, everything worked well.

[Tizull]

When using the sidecar method the plugin doesn't show up in the GUI as well.

[lixiaomei-git]

When using the sidecar method the plugin doesn't show up in the GUI as well.

@Tizull When using the sidecar method, the plugin does not show up in the GUI, which is expected behavior. Because argocd should identify the correct plugin by itself, instead of manually selecting plugins for the application. I think the sidecar method behaves like this: When the discover.find.command defined by sidecar is executed, and the file can be returned, this plugin will be applied.

[cschlesselmann]

Mind sharing the new error?

couldn't find cmp-server plugin supporting repository But I have executed the find command in the testapp directory, I can got the expected result.

/testapp$ find . -name '*.yaml' | xargs -I {} grep "<path\\|avp\\.kubernetes\\.io" {} | grep .
    avp.kubernetes.io/path: "kv/data/poc/argocd/example-secret"

And yesterday, I installed the plugin using argocd-cm configmap instead, everything worked well.

I am facing the same error currently and since running plugins via argocdm-cm is no longer supported i want to use the sidecar method. Did you stick with the argocd-cm method or have you discovered the problem with the sidecar approach?

Edit: Solved, my sidecar container wasn't running as UID 999