Support helm charts instead of kustomize as part of `bootstrap` command

[itai-codefresh]

[todaywasawesome]

The only thing keeping me from switching over all my management to autopilot.

[myspotontheweb]

Yes this would be cool.

I'm a big fan of how auto-pilot uses ApplicationSets, to support Kustomize deployments, but I too am a Helm person. I imagine this request would require changes to the auto-pilout "create project" command, which generates the underlying ApplicationSet, to be triggered based on the presence of different values files in the app chart directory

apps
  |-- myApp1
           |-- Chart.yaml
           |-- values-dev.yaml
           |-- values-test.yaml
           |-- values-prod.yaml

Unless anyone has a better idea on how to implement this?

[todaywasawesome]

@myspotontheweb For implementation I suggest sticking with kustomize and having it reference Helm. This is natively supported in Kustomize. See this comment for how to enable it with Argo CD.

This should work with autopilot as is, it's only the CLI commands that don't yet support it. So, create an app and update the kustomization to reference the Helm charts.

[patrickleet]

For autopilot, specifically, it can be added here:

bootstap/argo-cd/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
configMapGenerator:
- behavior: merge
  literals:
  - kustomize.buildOptions="--enable-alpha-plugins --enable-helm"
  - |
    repository.credentials=- passwordSecret:
        key: git_token
        name: autopilot-secret
      url: https://github.com/
      usernameSecret:
        key: git_username
        name: autopilot-secret
  name: argocd-cm
kind: Kustomization
namespace: argocd
resources:
- github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.3.6

And for completeness, here's a base prometheus deployment:

apps/prometheus/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmCharts:
- name: prometheus
  includeCRDs: true
  releaseName: prometheus
  version: 15.5.3
  repo: https://prometheus-community.github.io/helm-charts

[PG2000]

It seems this wouldn't work with OCI registries

https://github.com/kubernetes-sigs/kustomize/issues/4381

[Seji64]

For autopilot, specifically, it can be added here:

bootstap/argo-cd/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
configMapGenerator:
- behavior: merge
  literals:
  - kustomize.buildOptions="--enable-alpha-plugins --enable-helm"
  - |
    repository.credentials=- passwordSecret:
        key: git_token
        name: autopilot-secret
      url: https://github.com/
      usernameSecret:
        key: git_username
        name: autopilot-secret
  name: argocd-cm
kind: Kustomization
namespace: argocd
resources:
- github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.3.6

And for completeness, here's a base prometheus deployment:

apps/prometheus/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmCharts:
- name: prometheus
  includeCRDs: true
  releaseName: prometheus
  version: 15.5.3
  repo: https://prometheus-community.github.io/helm-charts

Is this still working in 0.4.7? I configured the argocd-cm config map but when i try to add an new app with the autopilot cli i receive the following error

INFO[2022-10-04T13:07:02Z] using revision: "", installation path: "/"
DEBU[2022-10-04T13:07:02Z] repository is ok
INFO[2022-10-04T13:07:02Z] inferred application type: kustomize
WARN[2022-10-04T13:07:02Z] using flat installation mode because base is a local file
INFO[2022-10-04T13:07:02Z] building manifests...
DEBU[2022-10-04T13:07:02Z] adjusting kustomization paths to local filesystem  from=/home/tim/traefik to=../traefik
DEBU[2022-10-04T13:07:02Z] running bootstrap kustomization: apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../traefik
  bootstrapKustPath=auto-pilot1406875415/kustomization.yaml resourcePath=../traefik
FATA[2022-10-04T13:07:02Z] failed to parse application from flags: failed running kustomization: accumulating resources: accumulation err='accumulating resources from '../traefik': '/home/tim/traefik' must resolve to a file': recursed accumulation of path '/home/tim/traefik': trouble configuring builtin HelmChartInflationGenerator with config: `
includeCRDs: true
name: traefik
releaseName: traefik/traefik
repo: https://helm.traefik.io/traefik
`: must specify --enable-helm

[patrickleet]

it is

> k get cm -n argocd argocd-cm -o yaml 

apiVersion: v1
data:
  kustomize.buildOptions: --enable-alpha-plugins --enable-helm
...

What's your cm look like?

[Seji64]

MNy argocd-cm looks like this:

kind: ConfigMap
apiVersion: v1
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: '--enable-helm'
  repository.credentials: |
    - passwordSecret:
        key: git_token
        name: autopilot-secret
      url: https://git.services.k-ops.io/
      usernameSecret:
        key: git_username
        name: autopilot-secret
  timeout.reconciliation: 15s

I also had the '--enable-alpha-plugins' configured but that did not change the behavior.

[dicolasi]

@Seji64 I can confirm that the approach described by @patrickleet works with that version. I just tried it now with minikube.

[NBonfattiMW]

pardon my ignorance, but I'm just getting started and all i have are helm charts. what do i need to do to create an app with argocd-autopilot for the prometheus example given?

I put the kustomization.yaml file in apps/prometheus/base and ran argocd-autopilot app create prometheus --app apps/prometheus/base --project my-project but I get the same error as Seji64.

my kubectl get cm -n argocd argocd-cm -o yaml output:

 →  kubectl get cm -n argocd argocd-cm -o yaml
apiVersion: v1
data:
  kustomize.buildOptions: --enable-alpha-plugins --enable-helm
  repository.credentials: |
    - passwordSecret:
        key: git_token        name: autopilot-secret
      url: https://github.com/
      usernameSecret:
        key: git_username
        name: autopilot-secret
  timeout.reconciliation: 15s
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"kustomize.buildOptions":"--enable-alpha-plugins --enable-helm","repository.credentials":"- passwordSecret:\n    key: git_token\n    name: autopilot-secret\n  url: https://github.com/\n  usernameSecret:\n    key: git_username\n    name: autopilot-secret\n","timeout.reconciliation":"15s"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"argo-cd","app.kubernetes.io/name":"argocd-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-cm","namespace":"argocd"}}
  creationTimestamp: "2023-02-01T19:00:44Z"
  labels:
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
  resourceVersion: "48305"
  uid: 5c4ef88f-2909-48c5-a7f0-1c892e8f4b7a

[patrickleet]

@NBonfattiMW

I was initially looking at the problem from this angle too, and here's what I've learned after understanding Argo more.

You don't need to use argocd-autopilot projects, which use kustomize, at all. You can just create a regular argocd appproject, and an application of other applications, one of which is a helm application.

Such as a "monitoring" appproject and application that include the prometheus application, which uses the regular argocd helm support.

If this doesn't make sense, read more argocd docs.

The argocd-autopilot projects approach is more tailored to DRYing the problem of replicating configs across many environments.

Now I just use argocd-autopilot to bootstrap and all my projects are traditional applications of applications, each in a separate repo. There's a bit more replication perhaps but I think different environments are distinct and important enough to be modeled separately anyway. 🤷‍♂️

There are use cases for both approaches, and you don't need to just use one or the other.

[NBonfattiMW]

ok but i still want to learn, and having the opinionated structure is helpful for me regardless. 😅

the error message @Seji64 and I saw about "must specify --enable-helm" is misleading, and it took awhile for the comment above "it's only the CLI commands that don't yet support it." to sink in. What I ended up doing was using autopilot to create the (non-helm) demo app in my project, and then replaced its base/kustomize with this one, and double checked that the app/prometheus/overlay/project/config.json for my project was sensible.

[Plork]

@myspotontheweb For implementation I suggest sticking with kustomize and having it reference Helm. This is natively supported in Kustomize. See this comment for how to enable it with Argo CD.

This should work with autopilot as is, it's only the CLI commands that don't yet support it. So, create an app and update the Kustomization to reference the Helm charts.

The only problem I have with this is that at the moment there is no way to provide the namespace for the helmcharts: in an overlay.

https://github.com/kubernetes-sigs/kustomize/issues/4303

So If I use the example from this tread when the chart uses release.Name or release.Namespace (as quite a lot do) this namespace gets inflated by Kustomize from helmcharts.*.release.Namespace. If you don't set this it will default to the namespace where Argo is installed.

I use argoCD applications or applicationsets to normally override the namespace via the ArgoCD applications and/or with Kustomize.Namespace and let a configurator in Kustomize change all references to the namespace. However the helmchart.*.ReleaseNamespace cannot be overwritten in an argoCD application thus needs to be hardcoded in every overlay.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmCharts:
- name: prometheus
  includeCRDs: true
  releaseName: prometheus
  **releaseNamespace: observability**
  version: 15.5.3
  repo: https://prometheus-community.github.io/helm-charts

namespace can be overwritten by ArgoCD but helmcharts.*.ReleaseNamespace cannot. This creates a double administration and the could create drift and misconfiguration.

Ideally I would like to have the base with a Kustomization for label pairs and some resource limits that might not be supported by the chart. Image overrides for private repos etc. Then in the overlay I just override some specifics and set a namespace (in case multiple environments use the same cluster, ie dev / test teams)

So If anyone knows how to solve this with Kustomize I would be very gratefull and otherwise the support for Helm ;)