Auto-deploying manifests: "helm repo add" does not work with corporate TLS certificates

[gbyczyns]

Is your feature request related to a problem? Please describe. My cluster is in a corporate network that captures all the HTTPS traffic and signs it with corporate TLS certificates coming from their own CA. I'd like to install Rancher by putting a HelmChart manifest into the /var/lib/rancher/k3s/server/manifests/ directory. The problem is that the repository uses the HTTPS scheme:

apiVersion: helm.cattle.io/v1
kind: HelmChart
spec:
  chart: rancher
  repo: https://releases.rancher.com/server-charts/latest
...

The auto-deployment feature creates a job that is unable to complete, because the related pod fails on certificate verification:

$ kubectl logs -n kube-system pod/helm-install-rancher-2rz7n
...
+ helm_v3 repo add rancher https://releases.rancher.com/server-charts/latest
Error: looks like "https://releases.rancher.com/server-charts/latest" is not a valid chart repository or cannot be reached: Get "https://releases.rancher.com/server-charts/latest/index.yaml": x509: certificate signed by unknown authority

Describe the solution you'd like I'd like to have a possibility to define a flag or an environmental variable during installation that will disable certificate verification for the auto-deployment feature. The helm repo add command already has several parameters that should fix this problem: --insecure-skip-tls-verify, --ca-file, --cert-file.

Describe alternatives you've considered I tried adding insecure-skip-tls-verify: true to /etc/rancher/k3s/k3s.yaml. I also tried running k3s kubectl config set-cluster default --insecure-skip-tls-verify=true. Did not help. The only alternative that works for me is to stop using the auto-deployment feature and instead run the helm command manually.

[brandond]

Adding those options to the admin kubeconfig (k3s.yaml) only changes the configuration that kubectl uses when connecting to the Kubernetes apiserver; it has no effect on the running cluster itself.

The K3s docs haven't been updated to reflect the new fields yet, but you can use the spec.repoCA field to add the certs used by your company's private CA. See https://github.com/k3s-io/helm-controller/pull/137#issue-1169033459 for more information on what this field should be set to.

[tuxpeople]

@brandond I assume it's the same Helm controller as it is in RKE2. At least the feature is there :-) Unfortunately, it isn't working.

The environment is Rancher 2.6.9, Kubernetes v1.23.15+rke2r1

I set spec.repoCA to the base64 encoded CA certificate. The controller spins up a pod to install the chart (using the image rancher/klipper-helm:v0.7.4-build20221121). The pod actually gets the certificate from the configmap and the helm argument is also being added:

helm_v3 repo add --ca-file /config/ca-file.pem cert-manager https://private-helmrepo/helm/chart-oss

But it fails:

Error: looks like "https://private-helmrepo/helm/chart-oss" is not a valid chart repository or cannot be reached: can't create TLS config for client: failed to append certificates from file: /config/ca-file.pem

I confirmed the following:

• its the correct CA certificate in the configmap
• I can manually use the very same command on a linux host as the pod and it works
• I used curl on on of the nodes to verify it's not an firewall issue by downloading https://private-helmrepo/helm/chart-oss/index.yaml

[brandond]

The error you're seeing is coming directly from the helm cli tool. How did you confirm that it is the correct file and has content usable by helm? Can you share what exactly your HelmChart manifest looks like? I suspect that the certificate is getting mangled somewhere along the way.

Note that as per the linked PR (emphasis added):

Adds support for spec.repoCA on the HelmChart CRD; if provided this is injected into the values ConfigMap and passed as the --ca-file flag when the Helm repo is added. Its contents should be a PEM-encoded CA certificate or certificate bundle, as expected by the helm CLI tools.

But you said:

I set spec.repoCA to the base64 encoded CA certificate.

This is not correct. The field needs to be set to a string containing the literal pem encoded certificate, pretty much the exact same way you set the valuesContent:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: grafana
  namespace: kube-system
spec:
  chart: stable/grafana
  targetNamespace: monitoring
  set:
    adminPassword: "NotVerySafePassword"
  valuesContent: |-
    image:
      tag: master
  repoCA: |-
    -----BEGIN CERTIFICATE-----
    MIIDoTCCAYmgAwIBAgIUF3lE2Ifd6kC/V1IP7NFtl/8AVyswDQYJKoZIhvcNAQEL
    BQAwKTEnMCUGA1UEAwweazNzLWludGVybWVkaWF0ZS1jYUAxNjcxMDY1MTI5MB4X
    DTIzMDExMzAwNDcwMVoXDTMzMDExMDAwNDcwMVowIzEhMB8GA1UEAwwYazNzLXNl

[tuxpeople]

@brandond I highly appreciate your help and you've been correct, it's my fault :-)

After fixing spec.repoCA it's now working as expected. Thanks!