pulumi-kubernetes
pulumi-kubernetes copied to clipboard
Published
20 hours ago •
pulumi
pulumi
Multiple Kustomize resources conflict
Hello!
- Vote on this issue by adding a 👍 reaction
- To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)
Issue details
Hi, I'm building an analytics stack on GKE. I need nginx-ingress-controller and cert-manager so I crafted overlays for both of them from official manifests. Specifically, I override cert-manager resources to deploy to its own namespace because base manifest tries to put things in kube-system (which is not allowed on autopilot clusters). The overlay deployment succeeds with kubectl apply -k. The overlay deployment succeeds in pulumi when there is no other Kustomize resources managed by pulumi. The overlay deployment fails in pulumi when there are other overlays scheduled after cert-manager (nginx for example and using dependsOn or not).
It looks like the rendered manifests lacks some fields, such as image name. I presume the pulumi-managed kustomize rendering gets confused when multiple directories are involved in the stack. This is difficult to debug because pulumi does not log the rendered manifests. I also do not use runtime transformation.
Steps to reproduce
Use multiple Kustomize local directories with overlays and deploy them in a pulumi stack. I'm running pulumi 3.22.1
Expected: A successful deployment Actual: A failed deployment
Sorry I do not provide much logs but if you could point me to a way to extract the manifests rendered by pulumi, I could attach more info. Thanks a lot !!
pulumi:pulumi:Stack analytics-infra-dev 2 messages
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-orders created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-webhook:subjectaccessreviews created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-clusterissuers created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-clusterissuers created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-webhook:subjectaccessreviews created
+ ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration cert-manager-webhook created
+ ├─ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration cert-manager-webhook created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-certificates created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager:leaderelection created
+ ├─ kubernetes:core/v1:Service cert-manager/cert-manager created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-cainjector created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-ingress-shim created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-challenges created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager:leaderelection created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-approve:cert-manager-io created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-edit created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-orders created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-issuers created
+ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager-webhook created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-ingress-shim created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-approve:cert-manager-io created
+ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager-cainjector created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager-webhook:dynamic-serving created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-view created
+ ├─ kubernetes:core/v1:Namespace cert-manager created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-certificates created
+ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager-webhook created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-challenges created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-certificatesigningrequests created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-issuers created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager-cainjector:leaderelection created
+ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager created
+ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager-cainjector created
+ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager-cainjector:leaderelection created
+ ├─ kubernetes:core/v1:Service cert-manager/cert-manager-webhook created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager-webhook:dynamic-serving created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-certificatesigningrequests created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-cainjector created
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition orders.acme.cert-manager.io created
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition certificaterequests.cert-manager.io created
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition certificates.cert-manager.io created
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition challenges.acme.cert-manager.io created
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition issuers.cert-manager.io created
+ └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition clusterissuers.cert-manager.io created
Diagnostics:
pulumi:pulumi:Stack (analytics-infra-dev):
Outputs:
Resources:
+ 46 created
23 unchanged
Duration: 2m11s
Deploy only cert-manager with pulumi => Success
pulumi:pulumi:Stack youstock-analytics-infra-dev running...
+ ├─ kubernetes:kustomize:Directory cert-manager-kustomize created
+ │ ├─ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration cert-manager-webhook creating
+ │ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager-webhook creating
+ │ ├─ kubernetes:core/v1:Service cert-manager/cert-manager-webhook creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-webhook:subjectaccessreviews creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-ingress-shim creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager-cainjector:leaderelection creating
+ │ ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration cert-manager-webhook creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-clusterissuers creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager-webhook:dynamic-serving creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-view creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-cainjector creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-orders creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-cainjector creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-certificates creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-challenges creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-clusterissuers creating
+ │ ├─ kubernetes:core/v1:Namespace cert-manager creating
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-webhook:subjectaccessreviews creating
+ │ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager-cainjector created
+ │ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager-webhook created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-edit created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager-webhook:dynamic-serving created
+ │ ├─ kubernetes:core/v1:ServiceAccount cert-manager/cert-manager created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-issuers created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager-cainjector:leaderelection created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-certificates created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-ingress-shim created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding cert-manager/cert-manager:leaderelection created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-issuers created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-orders created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role cert-manager/cert-manager:leaderelection created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-approve:cert-manager-io created
+ │ ├─ kubernetes:core/v1:Service cert-manager/cert-manager **creating failed** 1 error
+ │ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager-cainjector **creating failed** 1 error
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-approve:cert-manager-io created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-certificatesigningrequests created
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cert-manager-controller-certificatesigningrequests created
+ │ ├─ kubernetes:apps/v1:Deployment cert-manager/cert-manager **creating failed** 1 error
+ │ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole cert-manager-controller-challenges created
+ │ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition orders.acme.cert-manager.io created
+ │ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition certificaterequests.cert-manager.io created
+ │ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition certificates.cert-manager.io created
+ │ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition challenges.acme.cert-manager.io created
+ │ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition clusterissuers.cert-manager.io created
+ │ └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition issuers.cert-manager.io created
+ └─ kubernetes:kustomize:Directory nginx-kustomize created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding ingress-nginx created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role analytics/ingress-nginx creating
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding analytics/ingress-nginx creating
+ ├─ kubernetes:core/v1:ServiceAccount analytics/ingress-nginx-admission creating
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding ingress-nginx-admission created
+ ├─ kubernetes:core/v1:ServiceAccount analytics/ingress-nginx creating
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:Role analytics/ingress-nginx-admission creating
+ ├─ kubernetes:core/v1:ConfigMap analytics/ingress-nginx-controller creating
+ ├─ kubernetes:core/v1:Service analytics/ingress-nginx-controller-admission creating
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding analytics/ingress-nginx-admission creating
+ ├─ kubernetes:core/v1:Service analytics/ingress-nginx-controller creating
+ ├─ kubernetes:apps/v1:Deployment analytics/ingress-nginx-controller creating
+ ├─ kubernetes:batch/v1:Job analytics/ingress-nginx-admission-create creating
+ ├─ kubernetes:networking.k8s.io/v1:IngressClass ingress-nginx/nginx created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole ingress-nginx-admission created
+ ├─ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration ingress-nginx-admission created
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole ingress-nginx created
+ └─ kubernetes:batch/v1:Job analytics/ingress-nginx-admission-patch creating
^C
Diagnostics:
kubernetes:core/v1:Service (cert-manager/cert-manager):
error: 2 errors occurred:
* resource cert-manager/cert-manager was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: Resource operation was cancelled for "cert-manager"
* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
kubernetes:apps/v1:Deployment (cert-manager/cert-manager):
error: resource cert-manager/cert-manager was not successfully created by the Kubernetes API server : Deployment.apps "cert-manager" is invalid: spec.template.spec.containers[0].image: Required value
kubernetes:apps/v1:Deployment (cert-manager/cert-manager-cainjector):
error: resource cert-manager/cert-manager-cainjector was not successfully created by the Kubernetes API server : Deployment.apps "cert-manager-cainjector" is invalid: spec.template.spec.containers[0].image: Required value
Outputs:
Resources:
+ 50 created
17 unchanged
Duration: 2m39s
Deploy cert-manager and nginx => Fail
Hi thank for having a look, that was something like this:
const cert_manager_kustomize = new kubernetes.kustomize.Directory("cert-manager-kustomize", {
directory: "./overlay/cert-manager"
}, {
dependsOn: [ kubernetes_provider ],
provider: kubernetes_provider
});
const nginx_kustomize = new kubernetes.kustomize.Directory("nginx-kustomize", {
directory: "./overlay/nginx"
}, {
dependsOn: [ kubernetes_provider ],
provider: kubernetes_provider
});