fluxcd
HelmRepository (OCI) with insecure: true fails to pull from plain-http registry, attempts HTTPS first
Hi,
I am using in cluster harbor as the OCI registry to hold the helmchart which will be used by the HelmRepository. But chart failed to download form the plain http registry.
Environment
K8S: 1.30.11
Harbor Version: 2.7.1
Flux Version:
./flux version
flux: v2.6.4
distribution: flux-v2.6.4
helm-controller: v1.3.0
kustomize-controller: v1.6.1
source-controller: v1.6.2
Because the harbor running in cluster so I am using svc FQDN to access harbor OCI registry
The following are the svc of harbor which harbor-core svc using port 80
harbor-core ClusterIP 10.104.30.61 <none> 80/TCP 5d23h
harbor-database ClusterIP 10.110.87.163 <none> 5432/TCP 5d23h
harbor-jobservice ClusterIP 10.111.232.107 <none> 80/TCP 5d23h
harbor-portal ClusterIP 10.110.128.101 <none> 80/TCP 5d23h
harbor-redis ClusterIP 10.100.19.135 <none> 6379/TCP 5d23h
harbor-registry ClusterIP 10.106.89.164 <none> 5000/TCP,8080/TCP 5d23h
I did a test in a pod within the same cluster which use helm command directly login the harbor svc url and pull the chart which work as expected, the following commands were issued in a pod which installed the helm 3.17.3 (From the go.mod of the source-controller v1.6.2, the required helm version is 3.17.3, so I tested with the same version)
bash-4.2# ./helm version
version.BuildInfo{Version:"v3.17.3", GitCommit:"e4da49785aa6e6ee2b86efd5dd9e43400318262b", GitTreeState:"clean", GoVersion:"go1.23.7"}
bash-4.2# ./helm registry login harbor-core.cdp-system --insecure
Username: admin
Password:
Login Succeeded
bash-4.2# ./helm pull oci://harbor-core.cdp-system/cdp-foundation/charts/umami --version 4.2.0-3 --plain-http
Pulled: harbor-core.cdp-system/cdp-foundation/charts/umami:4.2.0-3
Digest: sha256:35b6e137280d6df9568869bec4eaff207c87949e66b59aef924371dc14ef20c3
bash-4.2# ls -lah
-rw-r--r-- 1 root root 156K Nov 17 11:49 umami-4.2.0-3.tgz
The following are the configuration of the helmrelease, and helmrepository
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: umami
namespace: cdp-foundation
spec:
dependsOn:
- name: sealed-secrets
namespace: cdp-foundation
interval: 2m0s
timeout: 10m0s
install:
remediation:
retries: -1
chart:
spec:
chart: umami
version: 4.2.0-3
sourceRef:
kind: HelmRepository
name: helm-repository
namespace: cdp-foundation
interval: 1m0s
values:
image:xxxx
xxxxxxx
HelmRepository
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: helm-repository
namespace: cdp-foundation
spec:
interval: 1m0s
timeout: 1m0s
type: oci
insecure: true
url: oci://harbor-core.cdp-system/cdp-foundation/charts
secretRef:
name: helm-harbor-secret
HelmRelease and HelmChart status
kubectl get hr umami -n cdp-foundation
NAME AGE READY STATUS
umami 60s False HelmChart 'cdp-foundation/cdp-foundation-umami' is not ready: latest generation of object has not been reconciled
kubectl get helmchart cdp-foundation-umami -n cdp-foundation
cdp-foundation-umami umami 4.2.0-3 HelmRepository helm-repository 2m32s False unknown build error: failed to login to OCI registry: Get "https://harbor-core.cdp-system/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Our testing shows that even with insecure: true, the source-controller still appears to first attempt a connection to the OCI repository URL using HTTPS (port 443). But the harbo-core svc only expose 80 port which lead to the connection timeout
This contrasts directly with the helm --plain-http command, which correctly forces an HTTP-only connection and works as expected. Because the source-controller attempts communication over both protocols, the configuration must allow it to resolve the hostname and access both port 443 and port 80 for it to eventually succeed over HTTP.
There is another issue which the HelmRepository not showing READY and STATUS
NAMESPACE NAME URL AGE READY STATUS
cdp-application helm-repository oci://harbor-core/cdp-application/charts 6d1h
cdp-foundation helm-repository oci://harbor-core.cdp-system/cdp-foundation/charts 6d1h
cdp-platform helm-repository oci://harbor-core/cdp-platform/charts 6d1h
cdp-system helm-repository oci://harbor-core/cdp-system/charts 6d1h
cdp-xaas helm-repository oci://haror-core.cdp-system 6d1h
Can you try OCIRepository and see if that works? Example here: https://fluxcd.io/flux/components/helm/helmreleases/#recommended-settings
@stefanprodan Yse I have tried OCIRepository, which worked with insecure: true setting.
kubectl get ocirepository -n cdp-xaas
NAME URL READY STATUS AGE
ocirepository oci://harbor-core.cdp-system/cdp-xaas/charts/adminer True stored artifact for digest '1.0.1@sha256:17de9348ed350089965064fa5c2938984d23e1e3cc3bbf5e264afec949caf3d5' 3d23h
kubectl get hr -n cdp-xaas
NAME AGE READY STATUS
adminer 3d23h True Helm install succeeded for release cdp-xaas/adminer.v1 with chart [email protected]+17de9348ed35
But for us it's hard to shift from helmrepository to ocirepository because of some legacy design.
Is there a previous version of Flux where this was working? (Is this a regression?)
But for us it's hard to shift from helmrepository to ocirepository because of some legacy design.
FYI new OCI-related features will only be added to OCIRepository. We're trying to make users switch from HelmRepository with type: oci to OCIRepository. Please add that to your tech debt backlog 😁
Is there a previous version of Flux where this was working? (Is this a regression?)
Currently our system is using flux 2.1.2, in 2.1.2 we have to do a lot of hacks add self-signed cert to the helmrepository to make it work. We want to remove the hacks, then we realized since the version 2.2.0 , HelmRepository start to support the field .spec.insecure.. Then for the new release we plan to upgrade flux to 2.3.0, but in 2.3.0 we got a lot of issues the for helmchart resource keep getting invalid host error then we decide to try the latest version before 2.7.0 which is the 2.6.4 then I got the issues above. So for the previous version never worked
Cool, if it never worked there's no regression 🙌
I suggest migrating to OCIRepository
