Helm repository not found

[iverberk]

Hi,

First of all, great job on the ArgoCD / Helmfile integration. It has been working very smoothly for us.

Up until now we were using local directories for the Helm chart sources. This works like a charm but we need to move the locations over to our internal Helm registry proxy.

I've added a repository definition as described in the docs:

repositories:
- name: internal-registry
  url: https://aaa.bbb.ccc

Next, in the release specification I use this repository for the Helm chart location:

releases:
  - name: example
    namespace: ns
    chart: internal-registry/example
    version: ~1.24.1 

When syncing this definition in ArgoCD it produces an error saying that Helm can't find the 'internal-registry' repository.

This repository needs to be added first before the template command can run successfully. In the source I noticed the --skip-deps argument: https://github.com/travisghansen/argo-cd-helmfile/blob/master/src/argo-cd-helmfile.sh#L408

Is this preventing the repository from being added? If so, how do we deal with this? If not, any suggestions why the repository might not be added before the template command?

Thanks!

[travisghansen]

Welcome! argocd plugins execute in 2 phases, an init phase and a generate phase. The repo side of it should be dealt with here: https://github.com/travisghansen/argo-cd-helmfile/blob/master/src/argo-cd-helmfile.sh#L353

[iverberk]

Thanks for the quick response.

Ah ok, so that clarifies where the repositories are supposed to be added. I'm only getting an error related to the generate command. ArgoCD doesn't seem to run the init command. I've on purpose modified my helmfile.yaml to produce an error when being processed by helmfile. I would expect ArgoCD to throw me an error that the init command failed because helmfile can't parse the helmfile.yaml file. Instead, I only again see the generate command.

Do you have any ideas why the init command might not be called or how to go about debugging?

[iverberk]

I've added a custom (non-existent) init script via the environment variable. It does fail with a message telling me that the script does not exist, so that leads me to conclude that init is being called. But why the helmfile repos command is not executed is still unclear to me.

[travisghansen]

If any command in the plugin fails the script will exit. So the behavior actually makes sense because the custom init script is executed just before the helmfile repos command.

[travisghansen]

Can you send over the application yaml and the version of the plugin currently in use? The only thing I can think of at the moment is there some logic error with the caching logic.

[iverberk]

Yeah I was also zooming in on the caching logic but couldn't find a good way to debug.

This is the application YAML (redacted):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: abc
spec:
  destination:
    namespace: ns
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps/abc
    plugin:
      env:
      - name: HELMFILE_TEMPLATE_OPTIONS
        value: --environment env-abc --include-crds
    repoURL: https://gitlab/aaa/bbb.git
    targetRevision: main
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=false

and the helmfile:

bases:
  - ../../../../env-dev.yaml
  - ../../../../env.yaml
---
repositories:
  - name: repo-abc
    url: https://aaa.bbb.ccc

releases:
  - name: dex
    namespace: dex
    chart: repo-abc/dex
    version: 0.14.1
    missingFileHandler: Warn
    values:
      - ./values.yaml
      {{- if .Values.dev  }}

      - ./overrides/dev/values.yaml
      - ./overrides/dev/{{ .Environment.Name }}/values.yaml

      {{- else }}

      - ./overrides/{{ .Environment.Name }}/values.yaml.gotmpl

      {{- end }}

  - name: manifests
    namespace: dex
    chart: ./manifests

Plugin version: travisghansen/argo-cd-helmfile:latest (unfortunately not pinned to a specific version). Is there any way to get more debug logging? To double check that the helmfile repos command is actually executed.

[travisghansen]

Set DEBUG=1 env var for the sidecar or in the app and it will spew a bunch out but it can be hard to find.

[iverberk]

As an intermediate finding I wanted to mention that creating a custom init script that calls the helmfile repos command actually solves the issue. This leads me to conclude that the helmfile repos command is not executed under normal circumstances. Just to validate: do you define repositories in the helmfile and do they work for you? This seems like the most basic of features so I would be surprised if it didn't work at all. I'll continue the investigation.

[travisghansen]

I do define the repos in my set of helmfiles. Something is clearly strange in your scenario. Maybe you can exec into the plugin container and try to execute the command directly and debug that way?

[iverberk]

Ok, I figured out why it was failing. My helmfile commands require an environment to be passed as argument otherwise it will fail. I added the HELMFILE_TEMPLATE_OPTIONS environment variable with the required environment setting. I didn't realise until now that I needed to set the HELMFILE_GLOBAL_OPTIONS environment variable so that the helmfile repos command also uses this argument. I never needed it before because the Helm charts were local to the repo.

I think what happens is, that the helmfile repos command silently fails without causing the script to error. Without an error you don't see any output and Argo CD will happily proceed to the generate stage. This construction might be a little bit too clever: https://github.com/travisghansen/argo-cd-helmfile/blob/master/src/argo-cd-helmfile.sh#L351

Shellcheck gives some information that might be useful: https://www.shellcheck.net/wiki/SC2015

Bash....so many subtle ways it might fail :-)

I'll leave it to you if you want me to close this issue or that you want to try out my theory regarding the silent failure and potentially fix it with a straight-up if-then-else construct.

[iverberk]

This little snippet clarifies things:

#!/bin/bash

set -e

true && {
  helmfile repos --tst
} || {
  echo "yo"
}

It will both produce an error because the --tst argument is not recognized and echo "yo". It will then exit with a zero exit code, indicating all went well which is obviously not what you want.

[travisghansen]

Yes! Thanks for pointing this out and great find! I'll get it cleaned up with something more appropriate.

[travisghansen]

Fix here: https://github.com/travisghansen/argo-cd-helmfile/commit/629949170a505e83ef21f40fe8ec69f452264460

Wait for v0.3.6 to build and then give it a try and let me know if it's fixed.