HelmChart doesn't support cross namespace sources

[steven-sheehy]

Since HelmRelease supports a CrossNamespaceObjectReference for SourceRef I thought I could create a single HelmRepository in the flux-system namespace and have multiple helm releases in different namespace share the same repository. This is more efficient and less likely to be rate limited than creating separate duplicate source repositories in each namespace.

When source-controller creates the HelmChart object from the HelmRelease and populates the SourceRef I thought it would match the same SourceRef from the HelmRelease. However, it seems it actually silently drops the namespace and then fails with failed to retrieve source: HelmRepository. I noticed that the HelmChart actually has a LocalHelmChartSourceReference so it doesn't support specifying a namespace. Since HelmRelease supports cross namespaces sources and source-controller is responsible for creating the HelmChart, I think HelmChart should allow cross namespace sources as well.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: hedera-mirror-node
  namespace: flux-system
spec:
  interval: 5m
  url: https://hashgraph.github.io/hedera-mirror-node/charts

apiVersion: v1
items:
- apiVersion: helm.toolkit.fluxcd.io/v2beta1
  kind: HelmRelease
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"helm.toolkit.fluxcd.io/v2beta1","kind":"HelmRelease","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"5b14dcac4dc782c21ac4848d860748d3fdf08519"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"mirror","namespace":"common"},"spec":{"chart":{"spec":{"chart":"hedera-mirror-common","sourceRef":{"kind":"HelmRepository","name":"hedera-mirror-node"},"version":"0.23.0-rc6"}},"install":{"crds":"CreateReplace"},"interval":"1m","timeout":"10m","upgrade":{"cleanupOnFail":true,"remediation":{"retries":10}},"values":{"prometheus":{"grafana":{"admin":{"existingSecret":""}},"prometheus":{"prometheusSpec":{"externalLabels":{"cluster":"preprod"}}},"prometheusOperator":{"namespaces":{"releaseNamespace":false}}},"traefik":{"service":{"annotations":{"cloud.google.com/neg":"{ \"exposed_ports\":{ \"443\":{\"name\": \"mirrornode-preprod\"} } }"}}}},"valuesFrom":[{"kind":"Secret","name":"mirror"}]}}
      kustomize.toolkit.fluxcd.io/checksum: 5b14dcac4dc782c21ac4848d860748d3fdf08519
    creationTimestamp: "2021-05-28T23:16:23Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 21
    labels:
      kustomize.toolkit.fluxcd.io/name: flux-system
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: mirror
    namespace: common
    resourceVersion: "26305355"
    uid: a405aff3-8851-4fad-83bb-c271bfa64419
  spec:
    chart:
      spec:
        chart: hedera-mirror-common
        sourceRef:
          kind: HelmRepository
          name: hedera-mirror-node
          namespace: flux-system
        version: 0.23.0-rc6
    install:
      crds: CreateReplace
    interval: 1m
    timeout: 10m
    upgrade:
      cleanupOnFail: true
      remediation:
        retries: 10
    values:
      prometheus:
        grafana:
          admin:
            existingSecret: ""
        prometheus:
          prometheusSpec:
            externalLabels:
              cluster: preprod
        prometheusOperator:
          namespaces:
            releaseNamespace: false
      traefik:
        service:
          annotations:
            cloud.google.com/neg: '{ "exposed_ports":{ "443":{"name": "mirrornode-preprod"}
              } }'
    valuesFrom:
    - kind: Secret
      name: mirror
  status:
    conditions:
    - lastTransitionTime: "2021-07-02T21:35:47Z"
      message: HelmChart 'common/common-mirror' is not ready
      reason: ArtifactFailed
      status: "False"
      type: Ready
    failures: 1
    helmChart: common/common-mirror
    lastAppliedRevision: 0.23.0-rc6
    lastAttemptedRevision: 0.23.0-rc6
    lastAttemptedValuesChecksum: fec25ec9ea83a228be2c7f48c4668dc06fdefa91
    lastReleaseRevision: 21
    observedGeneration: 21
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmChart
metadata:
  creationTimestamp: "2021-07-02T21:26:41Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  name: common-mirror
  namespace: flux-system
  resourceVersion: "26301479"
  uid: f886f8f3-5451-48b9-abc8-b207d1df9418
spec:
  chart: hedera-mirror-common
  interval: 1m0s
  sourceRef:
    kind: HelmRepository
    name: hedera-mirror-node
  version: 0.23.0-rc6
status:
  conditions:
  - lastTransitionTime: "2021-07-02T21:28:03Z"
    message: 'failed to retrieve source: HelmRepository.source.toolkit.fluxcd.io "hedera-mirror-node"
      not found'
    reason: ChartPullFailed
    status: "False"
    type: Ready
  observedGeneration: 1

kubectl get helmchart -A
NAMESPACE     NAME                 CHART                  VERSION      SOURCE KIND      SOURCE NAME          READY   STATUS                                                                                              AGE
flux-system   common-mirror        hedera-mirror-common   0.23.0-rc6   HelmRepository   hedera-mirror-node   False   failed to retrieve source: HelmRepository.source.toolkit.fluxcd.io "hedera-mirror-node" not found   52s
flux-system   performance-mirror   hedera-mirror          0.23.0-rc6   HelmRepository   hedera-mirror-node   False   failed to retrieve source: HelmRepository.source.toolkit.fluxcd.io "hedera-mirror-node" not found   52s

[JaneLiuL]

Hi @steven-sheehy
HelmRelease support crosss namepace sources, According to your status, it seems it could not found the hedera-mirror-node helm chart, could you upload the index file for check?

[steven-sheehy]

The index is here.

As I mentioned in the writeup, HelmRelease custom resource supports cross namespace sources but the HelmChart custom resource that is generated by Flux does not. You'll see in the docs that it uses a LocalHelmChartSourceReference for sourceRef.

[JaneLiuL]

@steven-sheehy
I just double check your helmrelease file, it seems your helmrelease format have issue, your kind is List, not the HelmRelease. Please correct as below format:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: mirror
spec:
  chart:
    spec:
      chart: hedera-mirror-common
      sourceRef:
        kind: HelmRepository
        name: hedera-mirror-node
        namespace: flux-system
      version: 0.23.0-rc6
  interval: 2m

Once correct your helmrelease file, it could fetch the helmchart as below:

And the helmrelease can be success.

[steven-sheehy]

It's a list because it's the output of kubectl get helmrelease -A -o yaml. I didn't specify the specific helmrelease name when I extracted that output, but it is correct in the cluster.

[JaneLiuL]

No, even we use kubectl get helmrelease -A -o yaml, the kind still be HelmRelease, the format must wrong for you. So please correct your format as I talk above.

[steven-sheehy]

Sorry, that's not correct. If you do a kubectl get <Kind> -o yaml without specifying the kind's metadata.name, the output will always be of type list. This List kind is documented as part of the output of kubectl. For example, here is the output when I list jobs (which I have zero):

~$ kubectl get job -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Here is the exact file in git being deployed to cluster that doesn't have the kubectl list wrapper (it's changed since I opened the issue and doesn't have the namespace in sourceRef).