pulumi-kubernetes icon indicating copy to clipboard operation
pulumi-kubernetes copied to clipboard

Increase retry count on Custom Resources waiting for CRDs

Open vyrwu opened this issue 3 years ago • 4 comments

When installing the kube-prometheus-stack chart with custom Prometheus Rules using v3.Helm library (TypeScript), some Custom Resources timeout with the following error:

authorizer.rules (iac:services:KubePrometheusStack$kubernetes:monitoring.coreos.com/v1:PrometheusRule)
Retry #0; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
Retry pulumi/pulumi#1; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
Retry pulumi/pulumi#2; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
Retry pulumi/pulumi#3; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
Retry pulumi/pulumi#4; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
Retry pulumi/pulumi#5; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
 
error: creation of resource cluster-services/authorizer.rules failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"

It seems like they can't find the corresponding CRD, which would be created and found if Pulumi just waited a few minutes, as there's a lot of different K8s resources being created as part of this deployment. I've tried adding a custom timeout of 30 minutes via a transformation on the Helm chart - however without success (the error log above was thrown after 7 minutes of build time):

transformations.push((obj: any, opts: CustomResourceOptions) => {
        if (obj.apiVersion === 'monitoring.coreos.com/v1' && obj.kind === 'PrometheusRule') {
            opts.customTimeouts = {
                create: '30m', <= Pulumi does not respect this in this scenario
            }
        }
    })

I want to tell Pulumi to keep retrying more than 5 times, because I know that the CRD will eventually be found. Does anyone have a suggestion on how to tackle this?

vyrwu avatar Jan 25 '21 16:01 vyrwu