pulumi-kubernetes
pulumi-kubernetes copied to clipboard
pulumi
Helm Chart Fails to Deploy
When deploying the GitLab Helm chart on EKS the deployment hangs. If I run the same chart manually from Helm directly, it works just fine.
Expected Behavior
GitLab Helm chart should deploy just like I can from Helm CLI.
Current Behavior
Pods fail to start, Pulumi hangs.
Steps to Reproduce
Try running this:
const gitlab = new k8s.helm.v3.Chart("gitlab-server", {
chart: "gitlab",
fetchOpts:{
repo: "https://charts.gitlab.io/",
},
values: {
"certmanager-issuer": {
email: "[email protected]",
},
global: {
hosts: {
domain: "mydomain.com",
},
},
}
});
Context (Environment)
I'm going to have to use Helm to install GitLab which is fine but ideally I keep all my IaC in one place. This also makes me hesitant to use Pulumi's Helm support because I'm unsure why this is failing.
I tried running the program you provided, and it also failed to deploy for me locally. I suspect this is a configuration issue, as I notice a lot of the Pods are showing unready, and one of the Jobs is also waiting with the following message:
Waiting for Job "gitlab-server-minio-create-buckets-1" to succeed (Active: 1 | Succeeded: 0 | Failed: 0)
I suspect this is related to the values provided to the chart, which look like default/dummy values. I'm not sure if you were using real values in your deployment?
Most likely, Pulumi is correctly detecting that these k8s resources are not ready, while Helm isn't actually checking the status of the resources it deploys.
I investigated a bit further here. Here's the equivalent Helm CLI command:
helm install gitlab gitlab \
--repo https://charts.gitlab.io/ \
--set [email protected] \
--set global.hosts.domain=mydomain.com
Looking at the status with kubectl, I found that some of the Pods are not coming up, despite Helm claiming that everything deployed successfully (which is based on the resources being created, not on checks to see that the resources are actually ready).
$ kubectl get po
NAME READY STATUS RESTARTS AGE
cm-acme-http-solver-ldkcb 1/1 Running 0 3m5s
cm-acme-http-solver-mfp5l 1/1 Running 0 3m5s
cm-acme-http-solver-sv5rc 1/1 Running 0 3m5s
gitlab-cainjector-67dbdcc896-kv6q2 1/1 Running 0 4m18s
gitlab-cert-manager-564fc9d7f5-bpgnj 1/1 Running 0 4m18s
gitlab-gitaly-0 0/1 Pending 0 4m17s
gitlab-gitlab-exporter-5c4b45cff8-64ggt 1/1 Running 0 4m17s
gitlab-gitlab-runner-68d65cc98c-6cn49 0/1 Running 1 4m16s
gitlab-gitlab-shell-df87cf6c-htngm 1/1 Running 0 4m18s
gitlab-gitlab-shell-df87cf6c-r8sdk 1/1 Running 0 3m17s
gitlab-issuer-1-5bxgk 0/1 Completed 0 4m17s
gitlab-migrations-1-5dd2c 0/1 CrashLoopBackOff 1 4m17s
gitlab-minio-56667f8cb4-lhqrb 1/1 Running 0 4m17s
gitlab-minio-create-buckets-1-sxpsj 0/1 Completed 0 4m17s
gitlab-nginx-ingress-controller-5d475855bb-6mwsb 1/1 Running 0 4m17s
gitlab-nginx-ingress-controller-5d475855bb-nmfxw 1/1 Running 0 4m17s
gitlab-nginx-ingress-default-backend-658cc89589-gvm2q 1/1 Running 0 4m18s
gitlab-postgresql-0 0/2 Pending 0 4m17s
gitlab-prometheus-server-768cd8f69-rdwfw 2/2 Running 0 4m17s
gitlab-redis-master-0 2/2 Running 0 4m17s
gitlab-registry-cb548b888-npwf9 1/1 Running 0 4m18s
gitlab-registry-cb548b888-vz44b 1/1 Running 0 4m18s
gitlab-sidekiq-all-in-1-v1-6b8b549f6-kv44s 0/1 Init:2/3 1 4m17s
gitlab-task-runner-6cfb8454c8-4fxtg 1/1 Running 0 4m17s
gitlab-webservice-default-76f4d669d7-wkzfv 0/2 Pending 0 4m16s
gitlab-webservice-default-76f4d669d7-zbl4m 0/2 Init:2/3 1 4m17s
Again, I suspect this has something to do with the configuration options, but I'm not familiar enough with this chart to diagnose right now.
@pauljrob Did you end up getting this working yet? The pod failures appear to be due to (at least) some of the issues noted in https://docs.gitlab.com/charts/installation/cloud/eks.html. These are ultimately issues/challenges with the GitLab Helm chart on EKS. I believe additional configuration of that chart is required to work correctly on EKS. I expect Pulumi can actually be really helpful here for scripting together the solutions to the issues noted there across both AWS and Kubernetes (provision storage volumes outside of the chart, doing the ACM provisioning and mapping that into a Kubernetes secret that the chart needs for manual cert management, and ultimately mapping the ELB hostname back into Route53).
It appears that the root of the problem is related to the GitLab configuration you're using, but here's a workaround to skip the readiness checks so that you get the same behavior as vanilla Helm (create the resources, but don't check for readiness):
const gitlab = new k8s.helm.v3.Chart("gitlab", {
chart: "gitlab/gitlab",
fetchOpts: {
repo: "https://charts.gitlab.io/",
},
values: {
"certmanager-issuer": {
email: `cameron@${domain_name}`,
},
global: {
hosts: {
domain: domain_name,
},
},
}
}, {
transformations: [
(args: pulumi.ResourceTransformationArgs) => {
if (args.props.metadata) {
return {
props: {
...args.props,
metadata: {
...args.props.metadata,
annotations: {
...args.props.metadata.annotations,
"pulumi.com/skipAwait": "true",
}
}
},
opts: args.opts,
}
}
return undefined;
},
]
});
We've opened the following issues to track improving this experience:
- https://github.com/pulumi/pulumi-kubernetes/issues/1417
- https://github.com/pulumi/pulumi-kubernetes/issues/1421
Since this issue is related to application configuration rather than Pulumi's k8s provider, I'm going to close this out now.
Since this issue is related to application configuration rather than Pulumi's k8s provider, I'm going to close this out now.
A chart written for Helm, which gets to define the rules of what is a legal Helm chart, could not be deployed by a default Pulumi configuration. That very much feels like a Pulumi problem to me.
