flux helmchart reconcile x509 issue

[cocampbe]

Describe the bug

I am in an on-premise k8s deployment. We also have azure devops on prem. We successfully deployed flux but are having issue with a specific chart being pulled. Part of the issue is that we have zscaler that proxies web traffic. For us to be able to pull data externally over https, we have to have the zscaler cert added to our cacerts. If not, it will not validate.

I added a helmrepo source and added the cafile. That worked pulling the index. But when a release is deployed and tried to pull the chart, I am getting the following error.

Get "https://github.com/airflow-helm/charts/releases/download/airflow-8.0.9/airflow-8.0.9.tgz": x509: certificate is valid for github.com, www.github.com, not airflow-helm.github.io

I think the issue is that I need the cacert for the zscaler used when pulling the chart. Otherwise, this issue is strange as I have seen the cert for this page and it has a SAN of *.github.io. Is there a way to add the cert for the helmchart?

I only seem to be having this issue with charts hosted on github.

Steps to reproduce

1. Be on-prem
2. have zscaler
3. Deploy flux
4. Punch self in face
5. add helm repo
6. deploy release

Expected behavior

No x509 issue

Screenshots and recordings

No response

OS / Distro

OL7u9

Flux version

0.16.2

Flux check

► checking prerequisites ✗ flux 0.16.2 <0.19.1 (new version is available, please upgrade) ✔ kubectl 1.19.6 >=1.18.0-0 ✔ Kubernetes 1.19.6 >=1.16.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.11.2 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.14.1 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.11.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.14.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.16.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.15.4 ✔ all checks passed

Git provider

ADO

Container Registry provider

No response

Additional context

2021-10-27T21:16:09.952Z error HelmChart/airflow-airflow.flux-system - Reconciler error Get "https://github.com/airflow-helm/charts/releases/download/airflow-8.0.9/airflow-8.0.9.tgz": x509: certificate is valid for github.com, www.github.com, not airflow-helm.github.io

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[makkes]

Just one thing that confuses me a little is that the URL is clearly https://github.com/... but the error says the cert isn't valid for airflow-helm.github.io. 🤔

[cocampbe]

@makkes The repo is https://airflow-helm.github.io/charts/. Likely just a redirect.

[cocampbe]

What is mildly interesting is that I think it still pulled the chart. I am not sure where the artifacts get stored. IN a perfect world, you would hope to not see an error. But if the artifact is pulled, then at least it is not critical.

[makkes]

@makkes The repo is https://airflow-helm.github.io/charts/. Likely just a redirect.

Even if it's a redirect, it should check the proper domain.

[hiddeco]

The HelmChart re-uses the authentication configuration of the HelmRepository to pull the chart.

The problem however seems to be that the index.yaml is hosted on github.io while it contains references to github.com. This raises the question to me if it might be related to security patches that were applied to Helm v3.6.1. In which case, enabling spec.passCredentials (at your own risk) might solve this.

[cocampbe]

@hiddeco I was thinking it was going to re-use the auth, but I did not want to assume that. I'll give your suggestion a try. We do have other options of putting the repo in git or artifactory.

[beandrad]

@cocampbe have you found a solution to this issue? I think we're facing a similar problem in our k3s cluster.

[cbutler-65]

I also have this problem. Is there a solution?

I am on-work-prem. I follow this guide https://fluxcd.io/docs/guides/sealed-secrets/

Error from source controller {"level":"debug","ts":"2022-05-30T07:46:40.298Z","logger":"events","msg":"Warning","object":{"kind":"HelmChart","namespace":"flux-system","name":"flux-system-sealed-secrets","uid":"23517ae7-0f1c-438d-bb6d-28823f7827ce","apiVersion":"source.toolkit.fluxcd.io/v1beta2","resourceVersion":"133036614"},"reason":"ChartPullError","message":"chart pull error: failed to download chart for remote reference: Get "https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.1.8/sealed-secrets-2.1.8.tgz": x509: certificate is valid for github.com, www.github.com, not bitnami-labs.github.io"}

Cause: Helm source points to: https://bitnami-labs.github.io/sealed-secrets/index.yaml

and this contains: urls: - https://github.com/bitnami-labs/sealed-secrets/releases/download/helm-v2.1.8/sealed-secrets-2.1.8.tgz version: 2.1.8

and something does not like it that helm source and release source are on different domains. https://bitnami-labs.github.io and https://github.com/bitnami-labs

any Ideas?

Thank You in advance Calum

[denniskniep]

Same Problem for External-Secrets Helm Chart: https://external-secrets.io/index.yaml contains https://github.com/external-secrets/external-secrets/releases/download/helm-chart-0.8.1/external-secrets-0.8.1.tgz

[mpizala]

We've same problem with https://hub.jupyter.org/helm-chart/ that contains https://jupyterhub.github.io/helm-chart/jupyterhub-2.0.0.tgz

[ricardopiteira]

I'm having the same problem ! Any workaround ?

[gberche-orange]

A potential workaround if the helm chart sources are available in a git repo is to reference the git repo instead, see https://fluxcd.io/flux/components/helm/helmreleases/#helm-chart-template

Given

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  namespace: fluxcd 
spec:
  interval: 5m
  url: https://charts.adfinis.com/
----
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: hedgedoc
spec:
  interval: 3m
  chart:
    spec:
      chart: hedgedoc
      version: 0.3.1
      sourceRef:
        kind: HelmRepository
        name: adfinis-charts-repo
        namespace: fluxcd 
      interval: 1m

with https://charts.adfinis.com/index.yaml returning

  hedgedoc:
  - annotations:
[...]
    urls:
    - https://github.com/adfinis/helm-charts/releases/download/hedgedoc-0.3.1/hedgedoc-0.3.1.tgz

and whose sources are available at https://github.com/adfinis/helm-charts/blob/hedgedoc-0.3.1/charts/hedgedoc

the workaround is then:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: adfinis-helmcharts-git-repo
spec:
  interval: 5m
  url: https://github.com/adfinis/helm-charts.git
  ref:
    tag: hedgedoc-0.3.1
  ignore: |
    # exclude all
    /*
    # include hedge doc only
    !/charts/hedgedoc/

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: hedgedoc

spec:
  interval: 3m
  chart:
    spec:
      chart: charts/hedgedoc
      version: 0.3.1
      sourceRef:
        kind: GitRepository
        name: adfinis-helmcharts-git-repo
      interval: 1m
  values:
[...]

[gberche-orange]

@hiddeco @stefanprodan any chance to move this issue to https://github.com/fluxcd/source-controller/ repo and rename it to something like HelmRepository fails to download urls from a distinct host than the one hosting index.yaml ?

Seems that the TLS certs are initialized once per helm repo in https://github.com/fluxcd/source-controller/blob/04612b539bc34d09348064f9896c10dc662a2e16/internal/controller/helmchart_controller.go#L517

Whereas they could likely be reinitialized before fetching an external referenced Url from a distinct FQDN (and hence distinct cert) at https://github.com/fluxcd/source-controller/blob/1aa9cf221983049861aa3190d407ab0dda07baae/internal/helm/repository/chart_repository.go#L277 ?

[stefanprodan]

I suggest using one of the various tools that can sync Helm charts from HTTP repos to your own container registry. IMO all Flux users should consider switching to Helm OCI.

See https://github.com/bitnami/charts-syncer