pulumi-kubernetes
pulumi-kubernetes copied to clipboard
Increase retry count on Custom Resources waiting for CRDs
When installing the kube-prometheus-stack chart with custom Prometheus Rules using v3.Helm library (TypeScript), some Custom Resources timeout with the following error:
authorizer.rules (iac:services:KubePrometheusStack$kubernetes:monitoring.coreos.com/v1:PrometheusRule)
Retry #0; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
Retry pulumi/pulumi#1; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
Retry pulumi/pulumi#2; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
Retry pulumi/pulumi#3; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
Retry pulumi/pulumi#4; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
Retry pulumi/pulumi#5; creation failed: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
error: creation of resource cluster-services/authorizer.rules failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
It seems like they can't find the corresponding CRD, which would be created and found if Pulumi just waited a few minutes, as there's a lot of different K8s resources being created as part of this deployment. I've tried adding a custom timeout of 30 minutes via a transformation on the Helm chart - however without success (the error log above was thrown after 7 minutes of build time):
transformations.push((obj: any, opts: CustomResourceOptions) => {
if (obj.apiVersion === 'monitoring.coreos.com/v1' && obj.kind === 'PrometheusRule') {
opts.customTimeouts = {
create: '30m', <= Pulumi does not respect this in this scenario
}
}
})
I want to tell Pulumi to keep retrying more than 5 times, because I know that the CRD will eventually be found. Does anyone have a suggestion on how to tackle this?