argoproj
helm chart dependencies fail to download from private repo
If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.
Checklist:
- [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- [X] I've included steps to reproduce the bug.
- [X] I've pasted the output of
argocd version.
Describe the bug
If using a private helm chart repo with an internally signed cert there does not seem to be away to use chart dependencies.
I can add a cert and the private repo and deploy a chart directly from the repo ok but if the chart includes a dependency it cannot download it during the helm dependency build command.
I'm following this guide https://www.openshift.com/blog/continuous-delivery-with-helm-and-argo-cd where i'd basically like to have a list of subcharts in git where I can bump the versions. This is because the deps are built and published by various third parties where I dont have the src as such.
time="2021-06-14T14:05:15Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in rd-release: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz: Get \"https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz\": x509: certificate signed by unknown authority" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-06-14T14:05:13Z" grpc.time_ms=2011.786 span.kind=server system=grpc
To Reproduce
Add a private chart repo and corresponding ca cert.
argocd cert add-tls core.harbor-${MINIKUBE_IP//./-}.nip.io --from /etc/pki/ca-trust/source/anchors/core.harbor-${MINIKUBE_IP//./-}.nip.io.crt
argocd repo add https://core.harbor-${MINIKUBE_IP//./-}.nip.io/chartrepo/example_helm --type helm --name example_helm --username admin --password ${HARBOR_ADMIN_PASSWORD}
Add a git repo with a chart like the following:
#Chart.yaml
apiVersion: v2
name: example-uber-release
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: 1.16.0
dependencies:
- name: example-app
version: 1.2.10
repository: https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm
Create an app and try to sync it.
Expected behavior
I would hope that both the login and tls cert would be used for the private repo but I guess this is maybe an edge case and argo isnt handling this. Is this supposed to work or is my approach unusual? I suppose I could get some real publicly signed certs but I'm trying a POC in minikube currently so didnt want to do that. I do wonder if it would then simply fail on authentication to the private repo anyways?
Version
(venv) [centos@tools-192-168-99-99 example-app]$ argocd version
argocd: v2.0.3+8d2b13d
BuildDate: 2021-05-27T17:38:37Z
GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
GitTreeState: clean
GoVersion: go1.16
Compiler: gc
Platform: linux/amd64
argocd-server: v2.0.3+8d2b13d
BuildDate: 2021-05-27T17:38:37Z
GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
GitTreeState: clean
GoVersion: go1.16
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
Helm Version: v3.5.1+g32c2223
Kubectl Version: v0.20.4
Jsonnet Version: v0.17.0
Logs
time="2021-06-14T14:05:15Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in rd-release: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz: Get \"https://core.harbor-192-168-99-125.nip.io/chartrepo/example_helm/charts/example-app-1.2.10.tgz\": x509: certificate signed by unknown authority" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-06-14T14:05:13Z" grpc.time_ms=2011.786 span.kind=server system=grpc
i had the same issue and found a workaround by mounting the ca-store of my kubernetes nodes which includes our root ca into the pods.
For example the with the following values of the argocd helm chart :
controller:
volumes:
- name: ca-certificates
hostPath:
path: /etc/ssl/certs/ca-bundle.crt
volumeMounts:
- name: ca-certificates
mountPath: /etc/ssl/certs/ca-certificates.crt
same for repoServer and Server.
we use centos7 on our hosts so your host path may vary.
I confirm the workaround described above works for https://github.com/argoproj/argo-cd/issues/13154 as well.
If you have an OCI registry for your helm chart, It appears that the reason the work around is needed is helm doesn't support adding oci repositories via "helm repo add"
Hence when argo uses: helm dependency build, helm attempts to download the oci helm chart but without a repo entry certs aren't used and it fails. If you do a helm pull, passing in the --ca-file option to helm it can successfully pull down the chart.
This actually appears a limitation of helm. Though looking at the documentation, a dependency update should work with oci registries (see https://helm.sh/docs/topics/registries/#specifying-dependencies) but the code doesn't support it with https://github.com/helm/helm/issues/12600 being the helm tracking task. Hence as of 5/Nov/23 the only work around is to override the container level CA so that openssl uses the system certs. There is no way to ignore the certs/override the certs in helm otherwise.
Instead of injecting the ca-bundle via ConfigMap/hostPath to /etc/ssl/certs/ca-certificates.crt, we've decided to build our own argo-cd image instead. This is our Dockerfile:
# syntax=docker/dockerfile:1.9
# image tag is automatically updated by renovate
FROM quay.io/argoproj/argocd:v2.12.2
# Changes to ca-certificates require root permissions
USER root
# Copy and install our custom root certificate
COPY --link company-ca.crt /usr/local/share/ca-certificates/company-ca.crt
RUN <<eol
#!/usr/bin/bash
set -e
chmod 644 /usr/local/share/ca-certificates/company-ca.crt
update-ca-certificates
eol
# Change back to the argocd user. $ARGOCD_USER_ID is defined in base image:
USER $ARGOCD_USER_ID
The whole process is automated using Renovate and automerge: true, so doing this for new versions of Argo CD is mostly painless. I then use Kustomize to patch the image references from the official installation manifests:
images:
# Use custom image with our root cert built-in:
- name: quay.io/argoproj/argocd
newName: our-internal-oci-repo.local/platform/ems-argo-cd
Instead of injecting the ca-bundle via ConfigMap/hostPath to
/etc/ssl/certs/ca-certificates.crt, we've decided to build our own argo-cd image instead. This is ourDockerfile:# syntax=docker/dockerfile:1.9 # image tag is automatically updated by renovate FROM quay.io/argoproj/argocd:v2.12.2 # Changes to ca-certificates require root permissions USER root # Copy and install our custom root certificate COPY --link company-ca.crt /usr/local/share/ca-certificates/company-ca.crt RUN <<eol #!/usr/bin/bash set -e chmod 644 /usr/local/share/ca-certificates/company-ca.crt update-ca-certificates eol # Change back to the argocd user. $ARGOCD_USER_ID is defined in base image: USER $ARGOCD_USER_IDThe whole process is automated using Renovate and
automerge: true, so doing this for new versions of Argo CD is mostly painless. I then use Kustomize to patch the image references from the official installation manifests:images: # Use custom image with our root cert built-in: - name: quay.io/argoproj/argocd newName: our-internal-oci-repo.local/platform/ems-argo-cd
That works for me. Thanks you for sharing it!.
