pulumi-kubernetes icon indicating copy to clipboard operation
pulumi-kubernetes copied to clipboard

Helm Release fails if Chart uses .Capabilities.APIVersions

Open LambArchie opened this issue 2 years ago • 0 comments

What happened?

I was trying to install Cilium's Helm Chart using Pulumi and once I enabled the ServiceMonitor option it failed, even though the cluster I was targeting had prometheus-operator CRDs installed in it.

When I investigated the chart I found it was using .Capabilities.APIVersions.Has to force it to error out if the CRD wasn't installed. The lines in the Helm file which are causing this issue are specified here https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/templates/validate.yaml#L18-L23.

I did find a closed issue in Cilium's repo which asked for a way to override this check but they closed the ticket as depending on the Kubernetes version and API versions available it changes what the chart outputs https://github.com/cilium/cilium/issues/16552

When I looked through Pulumi's code I saw that APIVersions is set to nil when passed into Helm for rendering https://github.com/pulumi/pulumi-kubernetes/blob/5277945f4d5fd63032d3f7014e27fb113335eeaa/provider/pkg/provider/helm_release.go#L1107

Therefore, I need a way to either manually specify what the APIVersions are or that Pulumi automatically detects the APIVersions installed in the target cluster and passes it into Helm.

Steps to reproduce

Created an empty stack and ensured that kubectl is pointing at a cluster with Prometheus Operator CRDs installed in it

import * as pulumi from '@pulumi/pulumi'
import * as k8s from '@pulumi/kubernetes'

const cilium = new k8s.helm.v3.Release('cilium', {
  chart: 'cilium',
  version: '1.12.0',
  repositoryOpts: {
    repo: 'https://helm.cilium.io/'
  },
  namespace: 'kube-system',
  values: {
    prometheus: {
      enabled: true,
      serviceMonitor: {
        enabled: true
      }
    }
  }
})

Expected Behavior

The helm release is deployed without error into the cluster which already contains the prometheus-operator CRD's

Actual Behavior

❯ pulumi preview
Previewing update (archie-test):
     Type                              Name                     Plan       Info
 +   pulumi:pulumi:Stack               archie-test-archie-test  create     
     └─ kubernetes:helm.sh/v3:Release  cilium                              1 error
 
Diagnostics:
  kubernetes:helm.sh/v3:Release (cilium):
    error: failed to create chart from template: execution error at (cilium/templates/validate.yaml:21:9): Service Monitor requires monitoring.coreos.com/v1 CRDs. Please refer to https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

Versions used

❯ pulumi about
CLI          
Version      3.37.2
Go Version   go1.18.4
Go Compiler  gc

Plugins
NAME        VERSION
kubernetes  3.20.4
nodejs      unknown

Host     
OS       darwin
Version  12.5
Arch     x86_64

This project is written in nodejs: executable='/Users/archie/.nvm/versions/node/v16.16.0/bin/node' version='v16.16.0'

Backend        
Name           ARCHIE-LAPTOP
URL            gs://pulumi-state
User           archie
Organizations  

Dependencies:
NAME                VERSION
@pulumi/pulumi      3.37.2
@types/node         14.18.24
@pulumi/kubernetes  3.20.4

Pulumi locates its logs in /var/folders/7f/4wh6789d7bl12zbm5kqf65m00000gp/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

I know the CRDs are installed correctly because kubectl reported they are created

❯ kubectl get customresourcedefinitions.apiextensions.k8s.io servicemonitors.monitoring.coreos.com
NAME                                    CREATED AT
servicemonitors.monitoring.coreos.com   2022-06-16T15:52:25Z

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

LambArchie avatar Aug 16 '22 16:08 LambArchie