flux2
flux2 copied to clipboard
fluxcd
Getting 401 using OCI HelmRepository with Azure Container Registry
Describe the bug
I have a MongoDB helm chart on Azure container registry as OCI artifacts. I can log in to the repository using
helm registry login my_domain.azurecr.io --username <my-username> --password <my-password>
and install the chart using
helm upgrade --install --namespace mongodb mongodb oci://my_domain.azurecr.io/helm/mongodb --version 10.30.3 --values my_values.yaml
But with the same credentials, I am getting 401 unauthorized while installing the chart with FluxCD OCI HelmRepository.
My HelmRepository and HelmRelease Object definition
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: mongodb-helm
namespace: mongodb
spec:
interval: 2m0s
url: oci://my_domain.azurecr.io/helm
type: "oci"
secretRef:
name: registry-token
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: mongodb
namespace: mongodb
spec:
targetNamespace: mongodb
releaseName: mongodb
interval: 2m
valuesFrom:
- kind: ConfigMap
name: mongodb-values
chart:
spec:
chart: mongodb
sourceRef:
kind: HelmRepository
name: mongodb-helm
version: '10.30.3'
Error message
{"level":"error","ts":"2022-07-11T08:50:41.748Z","logger":"controller.helmchart","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"mongodb","namespace":"mongodb","error":"chart pull error: chart pull error: failed to get chart version for remote reference: GET \"https://my_domain.azurecr.io/v2/helm/mongodb/tags/list\": unexpected status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
Steps to reproduce
Try to deploy using the objects mentioned in the description.
Expected behavior
Able to log in to Azure container registry and deploy an OCI helm chart.
Screenshots and recordings
No response
OS / Distro
Ubuntu 20.04
Flux version
flux: v0.31.3
Flux check
► checking prerequisites ✔ Kubernetes 1.22.10-eks-84b4fe6 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.22.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.26.2 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.24.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.25.9 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta1 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta1 ✔ receivers.notification.toolkit.fluxcd.io/v1beta1 ✔ all checks passed
Git provider
Azure DevOps
Container Registry provider
Azure DevOps
Additional context
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Hello,
can you provide kubect describe helmrepository output for you helmrepo?
Also please verify that your secret was generated without extra characters.
Thanks for your quick response, and sorry for the delayed reply. I cleaned the old setup and had to start from beginning to reproduce the issue.
Here's the output you asked for
❯ kubectl describe helmrepository
No resources found in default namespace.
❯ kubectl describe helmrepository -n mongodb
Name: mongodb-helm
Namespace: mongodb
Labels: kustomize.toolkit.fluxcd.io/name=mongodb
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: reconcile.fluxcd.io/requestedAt: 2022-07-20T22:20:15.106814+02:00
API Version: source.toolkit.fluxcd.io/v1beta2
Kind: HelmRepository
Metadata:
Creation Timestamp: 2022-07-20T19:07:20Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Managed Fields:
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:kustomize.toolkit.fluxcd.io/name:
f:kustomize.toolkit.fluxcd.io/namespace:
f:spec:
f:interval:
f:secretRef:
f:name:
f:type:
f:url:
Manager: kustomize-controller
Operation: Apply
Time: 2022-07-20T19:07:20Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: source-controller
Operation: Update
Time: 2022-07-20T19:07:20Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:reconcile.fluxcd.io/requestedAt:
Manager: flux
Operation: Update
Time: 2022-07-20T20:20:15Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:lastHandledReconcileAt:
f:observedGeneration:
Manager: source-controller
Operation: Update
Subresource: status
Time: 2022-07-20T20:20:19Z
Resource Version: 19992776
UID: bebb1b70-4b6c-4a34-b0c6-5a55274b172f
Spec:
Interval: 2m0s
Secret Ref:
Name: registry-token
Timeout: 60s
Type: oci
URL: oci://my_domain.azurecr.io/helm
Status:
Conditions:
Last Transition Time: 2022-07-20T20:11:58Z
Message: Helm repository is ready
Observed Generation: 1
Reason: Succeeded
Status: True
Type: Ready
Last Handled Reconcile At: 2022-07-20T22:20:15.106814+02:00
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Succeeded 11m source-controller Helm repository is ready
I think the secret is fine, as I can pull images from the same registry using the same secret.
yes the secret seems fine.
The issue is the .Spec.URL. You have oci://my_domain.azurecr.io/helm in your Helmreposiroty and I think it should be oci://my_domain.azurecr.io/helm/mongodb. Because I think you have a mongodb chart in helm/mongodb/.
I actually did this at the beginning, and it didn't work. Then I saw in issue#2900 it was suggested to remove the last part from the URL. So I did that as well. But the result is always 401.
Here's the current log from the source controller
{"level":"error","ts":"2022-07-21T18:34:48.090Z","logger":"controller.helmchart","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"mongodb-mongodb","namespace":"mongodb","error":"chart pull error: chart pull error: failed to get chart version for remote reference: GET \"https://my_domain.azurecr.io/v2/helm/mongodb/mongodb/tags/list\": unexpected status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
And
❯ kubectl describe helmrepository -n mongodb
Name: mongodb-helm
Namespace: mongodb
Labels: kustomize.toolkit.fluxcd.io/name=mongodb
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: reconcile.fluxcd.io/requestedAt: 2022-07-21T18:04:30.364482+02:00
API Version: source.toolkit.fluxcd.io/v1beta2
Kind: HelmRepository
Metadata:
Creation Timestamp: 2022-07-20T19:07:20Z
Finalizers:
finalizers.fluxcd.io
Generation: 2
Managed Fields:
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:kustomize.toolkit.fluxcd.io/name:
f:kustomize.toolkit.fluxcd.io/namespace:
f:spec:
f:interval:
f:secretRef:
f:name:
f:type:
f:url:
Manager: kustomize-controller
Operation: Apply
Time: 2022-07-21T15:59:04Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: source-controller
Operation: Update
Time: 2022-07-20T19:07:20Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:reconcile.fluxcd.io/requestedAt:
Manager: flux
Operation: Update
Time: 2022-07-20T20:20:15Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:lastHandledReconcileAt:
f:observedGeneration:
Manager: source-controller
Operation: Update
Subresource: status
Time: 2022-07-20T20:20:19Z
Resource Version: 20401470
UID: bebb1b70-4b6c-4a34-b0c6-5a55274b172f
Spec:
Interval: 2m0s
Secret Ref:
Name: registry-token
Timeout: 60s
Type: oci
URL: oci://my_domain.azurecr.io/helm/mongodb
Status:
Conditions:
Last Transition Time: 2022-07-20T20:11:58Z
Message: Helm repository is ready
Observed Generation: 2
Reason: Succeeded
Status: True
Type: Ready
Last Handled Reconcile At: 2022-07-21T18:04:30.364482+02:00
Observed Generation: 2
Events: <none>
Previously, without chart name at the end of the OCI URL flux was trying to reach
https://my_domain.azurecr.io/v2/helm/mongodb/tags/list
After adding the chart name at the end, it is now trying to reach https://my_domain.azurecr.io/v2/helm/mongodb/mongodb/tags/list, this isn't correct, right?
Hi @ahsanemon sorry for misleading you. https://my_domain.azurecr.io/v2/helm/mongodb/tags/list is right.
I tested it, and here is what I get
Name: podinfo-private-az
Namespace: default
Labels: <none>
Annotations: <none>
API Version: source.toolkit.fluxcd.io/v1beta2
Kind: HelmRepository
Metadata:
Creation Timestamp: 2022-07-21T08:07:21Z
Finalizers:
finalizers.fluxcd.io
Generation: 2
Managed Fields:
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: source-controller
Operation: Update
Time: 2022-07-21T08:07:21Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:interval:
f:secretRef:
.:
f:name:
f:timeout:
f:type:
f:url:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-21T08:07:59Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:observedGeneration:
Manager: source-controller
Operation: Update
Subresource: status
Time: 2022-07-21T08:09:00Z
Resource Version: 221490
UID: 513dc681-e514-4846-ba27-8c85c902cf61
Spec:
Interval: 1m0s
Secret Ref:
Name: azregcred
Timeout: 60s
Type: oci
URL: oci://***.azurecr.io/charts
Status:
Conditions:
Last Transition Time: 2022-07-21T22:24:34Z
Message: Helm repository is ready
Observed Generation: 2
Reason: Succeeded
Status: True
Type: Ready
Observed Generation: 2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Succeeded 24m source-controller Helm repository is ready
Name: podinfo-azure
Namespace: default
Labels: <none>
Annotations: <none>
API Version: helm.toolkit.fluxcd.io/v2beta1
Kind: HelmRelease
Metadata:
Creation Timestamp: 2022-07-21T08:09:15Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Managed Fields:
API Version: helm.toolkit.fluxcd.io/v2beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: helm-controller
Operation: Update
Time: 2022-07-21T08:09:15Z
API Version: helm.toolkit.fluxcd.io/v2beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:chart:
.:
f:spec:
.:
f:chart:
f:reconcileStrategy:
f:sourceRef:
.:
f:kind:
f:name:
f:version:
f:interval:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-21T08:09:15Z
API Version: helm.toolkit.fluxcd.io/v2beta1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:helmChart:
f:lastAppliedRevision:
f:lastAttemptedRevision:
f:lastAttemptedValuesChecksum:
f:lastReleaseRevision:
f:observedGeneration:
Manager: helm-controller
Operation: Update
Subresource: status
Time: 2022-07-21T22:24:56Z
Resource Version: 221645
UID: 88514f0a-1e66-4a05-b59b-24bad69e5b2a
Spec:
Chart:
Spec:
Chart: podinfo
Reconcile Strategy: ChartVersion
Source Ref:
Kind: HelmRepository
Name: podinfo-private-az
Version: >=6.1.0
Interval: 1m0s
Status:
Conditions:
Last Transition Time: 2022-07-21T22:24:56Z
Message: Release reconciliation succeeded
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Last Transition Time: 2022-07-21T08:09:18Z
Message: Helm install succeeded
Reason: InstallSucceeded
Status: True
Type: Released
Helm Chart: default/default-podinfo-azure
Last Applied Revision: 6.1.4
Last Attempted Revision: 6.1.4
Last Attempted Values Checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
Last Release Revision: 1
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 26m helm-controller HelmChart 'default/default-podinfo-azure' is not ready
Name: default-podinfo-azure
Namespace: default
Labels: <none>
Annotations: <none>
API Version: source.toolkit.fluxcd.io/v1beta2
Kind: HelmChart
Metadata:
Creation Timestamp: 2022-07-21T08:09:15Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Managed Fields:
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:chart:
f:interval:
f:reconcileStrategy:
f:sourceRef:
.:
f:kind:
f:name:
f:version:
Manager: helm-controller
Operation: Update
Time: 2022-07-21T08:09:15Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: source-controller
Operation: Update
Time: 2022-07-21T08:09:15Z
API Version: source.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
f:artifact:
.:
f:checksum:
f:lastUpdateTime:
f:path:
f:revision:
f:size:
f:url:
f:conditions:
f:observedChartName:
f:observedGeneration:
f:url:
Manager: source-controller
Operation: Update
Subresource: status
Time: 2022-07-21T08:09:17Z
Resource Version: 221493
UID: c73afe3f-eb5e-4a4f-8e69-2a88c7cdf007
Spec:
Chart: podinfo
Interval: 1m0s
Reconcile Strategy: ChartVersion
Source Ref:
Kind: HelmRepository
Name: podinfo-private-az
Version: >=6.1.0
Status:
Artifact:
Checksum: 63f416297202c14e8b0c2ddafbcc153b67ede5fa252e7e67299d22dca5e6dd7b
Last Update Time: 2022-07-21T08:09:17Z
Path: helmchart/default/default-podinfo-azure/podinfo-6.1.4.tgz
Revision: 6.1.4
Size: 13548
URL: http://source-controller.flux-system.svc.cluster.local./helmchart/default/default-podinfo-azure/podinfo-6.1.4.tgz
Conditions:
Last Transition Time: 2022-07-21T22:24:34Z
Message: pulled 'podinfo' chart with version '6.1.4'
Observed Generation: 1
Reason: ChartPullSucceeded
Status: True
Type: Ready
Last Transition Time: 2022-07-21T08:09:17Z
Message: pulled 'podinfo' chart with version '6.1.4'
Observed Generation: 1
Reason: ChartPullSucceeded
Status: True
Type: ArtifactInStorage
Observed Chart Name: podinfo
Observed Generation: 1
URL: http://source-controller.flux-system.svc.cluster.local./helmchart/default/default-podinfo-azure/latest.tar.gz
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ChartPullSucceeded 27m source-controller pulled 'podinfo' chart with version '6.1.4'
Normal ArtifactUpToDate 29s (x28 over 27m) source-controller artifact up-to-date with remote revision: '6.1.4'
my path is oci://***.azurecr.io/charts/podinfo:6.1.4
I am getting the same error even after switching from oci://my_domain.azurecr.io/helm to oci://my_domain.azurecr.io/charts.
Although I am using static token to access the ACR and as I mentioned previously that I can pull image using this token without any issue, but I have MFA enabled for Azure access. Is it possible by any chance that fluxCD is hitting any Azure endpoint that is behind MFA?
@souleb we face the same issue:
"error":"chart pull error: chart pull error: failed to get chart version for remote reference: GET \"https://ACR_NAME.azurecr.io/v2/helm/test-chart/tags/list\": unexpected status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
The same works however with helm registry login and helm install using the same credentials
it seems controller.helmchart doesn't do proper authentication and does not obtain the tocken which helm registry login does
Flux 0.33 comes with support for Azure auto-login, instead of using a secret and static credentials, Flux is now able to use aad-pod-identity. Docs here: https://fluxcd.io/docs/components/source/helmrepositories/#azure
