Feedback about lovely plugin trial

[rcarre]

Dear all, I am working in MonoRepo<=>Multi-k8s-clusters environment. Here is my inventory repository structure my GitlabCI pipeline is browsing to build deployment manifests to be scrapped by Argo App.

I had in mind to move the process of my GitlabCI pipeline into Lovely plugin. Here are the main caveats I do experience

Vault interop

Vault Replacer plugin is configured as a plugin to lovely by the means of env variable ARGOCD_ENV_LOVELY_PLUGINS. As a result, Vault Replacer is gonna run in preprocessing step in order to resolve secret in these cases:

Deploy a helm chart in accordance with its values. One of the values will be taken from Hashicorp Vault.
Deploy a separate kubernetes secret, with the value being taken from Hashicorp Vault.
Deploy a kubernetes configmap using Kustomize. Some of the data within the configmap will be taken from Hashicorp Vault.

Vault Replacer does only support Kubernetes Auth method. This method is not supported by my central Vault server. As a workaround, I successfully interop lovely plugin with ArgoCD Vault plugin with approle authentication. ArgoCD Vault plugin takes the PATH of the yaml files to resolve and sends the result as the aggregate standard output i.e. we should iterate each yaml file to keep the directory tree like Vault Replacer which performs this process in the box. Vault plugin cannot do it out of the box along preprocessing step by the means of the basic ArgoCD configManagementPlugins syntax.

As a consequence, I trigger Vault Plugin once Lovely Plugin is processed. Hence, we support only here above use cases #2 and #3.

Overlays support

As detailed in ArgoCD Lovely plugin help page, overlays mechanisms are not supported:

• neither from an Helm value file perspective : --values (or -f) to specify a YAML file with overrides. This can be specified multiple times and the rightmost file will take precedence,
• nor from a Kustomize perspective.

However, application sets should be investigated furthermore to mitigate/tackle this caveat.

Environment variables

ArgoCD Lovely plugin does not support environnement variables. Helm value file or manifest file must be fully resolved prior to be processed. Also ArgoCD Lovely plugin does not enhance Kustomize processing which does build the manifests from files exclusively, and no information would come from runtime. In essence, no information can be loaded from the command-line arguments or environment variables during the manifest build phase.

This caveat could however be solved by the means of a preprocessor plugin which would browse the working application directory and envsubst the yaml files.

Thanks & Best Regards. Richard

[Joibel]

Vault Interop Regarding ArgoCD Vault Plugin: This was one of the reasons for writing vault-replacer. Deploying a helm chart with values.yaml containing a replacable thing doesn't work in ArgoCD vault plugin, and cannot be made to. Adding approle support to vault-replacer would be my fix for your scenario if I understand it correctly. Overlays Extending helm support in lovely for overlays would be easy and I'm happy to do that. Kustomize overlays should already work, you just specify the location of the overlay. What doesn't work will be local relative paths outside of the current working directory, but this is because of Argo CD sandboxing of application deployments to prevent viewing of other application data. If you refer to a remote base it should work (although we don't support authentication at the moment (see #36). Environment variables I'm not sure what environment variables you want to do what. (Names are restricted by Argo CD (see #34)) The documented environment variables for merge or patching values.yaml or kustomization.yaml may be able to help you. Please could you document what you'd like to see happen here?

[rcarre]

Thanks @Joibel for your swift reply.

About Vault interop, I do confirm that adding approle support to vault-replacer is ok.

About overlays, you highlighted what's wrong in my case : What doesn't work will be local relative paths outside of the current working directory. In order to avoid duplicated configuration, instances configuration must be always setup as a deviation from a base setup. From this Top => Down model, I don't see how to manage all in a current working directory ...

About environment variables, this is the same idea : avoid hard coded duplicated configuration. Please have a look to the example below. Let's consider an Ingress to access Prometheus GUI. The ingress manifest file is managed as a kustomize extra resource to Prometheus Helm chart. The Ingress.spec.rules.host value does include the ${WILDCARD_FQDN} path of the k8s cluster where the ingress is gonna be deployed. What's the way to resolve such environment variable along Lovely plugin processing path ?

• Dirty & short term solution

From ArgoCD 2.4, rename WILDCARD_FQDN into ARGOCD_ENV_WILDCARD_FQDN In generate plugin command, add a first step to browse & envsubst the current working directory and then run lovely plugin

• Mid term solution

As vault-replacer plugin does, build an envsubst-replacer running as preprocessing plugin of lovely plugin. Plugin envsubs-replacer should get the environment variable from a file in the current working directory.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-signin: "https://oauth2.${WILDCARD_FQDN}/oauth2/start"
    nginx.ingress.kubernetes.io/auth-url: "http://${ARGOCD_OAUTH2_APP_NAME}-oauth2-proxy.${ARGOCD_OAUTH2_APP_NAMESPACE}.svc.cluster.local/oauth2/auth"
  name: "${ARGOCD_APP_NAME}-prometheus-oauth2"
spec:
  rules:
  - host: "${ARGOCD_APP_NAME}-prometheus-oauth2.${WILDCARD_FQDN}"
    http:
      paths:
      - backend:
          service:
            name: "${ARGOCD_APP_NAME}-prometheus-prometheus"
            port:
              number: 9090
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - "${ARGOCD_APP_NAME}-prometheus-oauth2.${WILDCARD_FQDN}"
    secretName: "${ARGOCD_APP_NAME}-tls-secret"

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.