pulumi-kubernetes
pulumi-kubernetes copied to clipboard
Published
20 hours ago •
pulumi
pulumi
Missing resource when using `k8s.helm.v3.Chart` to deploy a Chart
What happened?
Deploying the chart for actions-runner-controller I found an odd behavior with the Chart deployment. When using new k8s.helm.v3.Chart with fetchOpts it misses the Ingress resource, however, cloning the mentioned chart locally setting the same tag of release, and replacing the fetchOpts to path works - I get the Ingress resource to be created.
Plus, using new k8s.helm.v3.Release instead of new k8s.helm.v3.Chart with the same options just changing the fetchOpts to its respective equivalent repositoryOpts it also works - Ingress resource is there to be created.
Steps to reproduce
Deploy the chart for actions-runner-controller using the following configuration:
const chart = new k8s.helm.v3.Chart(product, {
chart: 'actions-runner-controller',
version: '0.20.2',
fetchOpts: {
repo: 'https://actions-runner-controller.github.io/actions-runner-controller',
},
namespace: namespace.metadata.name,
values: {
replicaCount: 2,
certManagerEnabled: false,
authSecret: {
create: true,
github_token: configurations.requireSecret('token'),
},
githubWebhookServer: {
enabled: true,
ingress: {
enabled: true,
ingressClassName: `nginx-public`,
hosts: [
{
host: `${product}.${domain}`,
paths: [
{
path: '/actions-runner-controller-github-webhook-server',
pathType: 'Prefix',
},
],
},
],
},
},
},
});
You won't get the Ingress resource deployed:
$ kubectl get --namespace github all
NAME READY STATUS RESTARTS AGE
pod/github-actions-runner-controller-5bdd6dc4bc-2vx48 2/2 Running 0 22h
pod/github-actions-runner-controller-5bdd6dc4bc-wvzvd 2/2 Running 0 22h
pod/github-actions-runner-controller-github-webhook-server-6f65r6w4 2/2 Running 0 22h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/github-actions-runner-controller-github-webhook-server ClusterIP 172.20.20.84 <none> 80/TCP 14d
service/github-actions-runner-controller-metrics-service ClusterIP 172.20.33.248 <none> 8443/TCP 15d
service/github-actions-runner-controller-webhook ClusterIP 172.20.24.170 <none> 443/TCP 15d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/github-actions-runner-controller 2/2 2 2 15d
deployment.apps/github-actions-runner-controller-github-webhook-server 1/1 1 1 14d
NAME DESIRED CURRENT READY AGE
replicaset.apps/github-actions-runner-controller-5bdd6dc4bc 2 2 2 15d
replicaset.apps/github-actions-runner-controller-7c48789b 0 0 0 13d
replicaset.apps/github-actions-runner-controller-bf9cfcb6 0 0 0 14d
replicaset.apps/github-actions-runner-controller-github-webhook-server-6f67fbcd6d 1 1 1 14d
replicaset.apps/github-actions-runner-controller-github-webhook-server-74d97f498f 0 0 0 14d
$ pulumi up
Previewing update (staging)
View Live: <uri>
Type Name Plan
pulumi:pulumi:Stack git-staging
Resources:
34 unchanged
Do the same deployment but instead of fetching from the repo, clone the mentioned chart locally, check it out to the same version, set up the path option, and run pulumi up the resource will be created:
const chart = new k8s.helm.v3.Chart(product, {
chart: 'actions-runner-controller',
version: '0.20.2',
path: '../../../actions-runner-controller/charts',
namespace: namespace.metadata.name,
values: {
replicaCount: 2,
certManagerEnabled: false,
authSecret: {
create: true,
github_token: configurations.requireSecret('token'),
},
githubWebhookServer: {
enabled: true,
ingress: {
enabled: true,
ingressClassName: `nginx-public`,
hosts: [
{
host: `${product}.${domain}`,
paths: [
{
path: '/actions-runner-controller-github-webhook-server',
pathType: 'Prefix',
},
],
},
],
},
},
},
});
$ pulumi up
Previewing update (staging)
View Live: <uri>
Type Name Plan
pulumi:pulumi:Stack git-staging
└─ kubernetes:helm.sh/v3:Chart github
+ └─ kubernetes:networking.k8s.io/v1:Ingress github/github-actions-runner-controller-github-webhook-server create
Resources:
+ 1 to create
34 unchanged
Or, using k8s.helm.v3.Release instead of k8s.helm.v3.Chart also creates the Ingress resource:
const chart = new k8s.helm.v3.Release(product, {
chart: 'actions-runner-controller',
version: '0.20.2',
repositoryOpts: {
repo: 'https://actions-runner-controller.github.io/actions-runner-controller',
},
namespace: namespace.metadata.name,
values: {
replicaCount: 2,
certManagerEnabled: false,
authSecret: {
create: true,
github_token: configurations.requireSecret('token'),
},
githubWebhookServer: {
enabled: true,
ingress: {
enabled: true,
ingressClassName: `nginx-public`,
hosts: [
{
host: `${product}.${domain}`,
paths: [
{
path: '/actions-runner-controller-github-webhook-server',
pathType: 'Prefix',
},
],
},
],
},
},
},
});
$ pulumi up
...
+ kubernetes:helm.sh/v3:Release: (create)
[urn=urn:pulumi:staging::git::kubernetes:helm.sh/v3:Release::github]
[provider=urn:pulumi:staging::git::pulumi:providers:kubernetes::default_3_21_0::d2fedf5e-88c2-4235-b119-15307abdebda]
atomic : false
chart : "actions-runner-controller"
cleanupOnFail : false
createNamespace : false
dependencyUpdate : false
devel : false
disableCRDHooks : false
disableOpenapiValidation: false
disableWebhooks : false
forceUpdate : false
lint : false
name : "github-ee486465"
namespace : "github"
recreatePods : false
renderSubchartNotes : false
replace : false
repositoryOpts : {
repo : "https://actions-runner-controller.github.io/actions-runner-controller"
}
resetValues : false
resourceNames : {
ClusterRole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1 : [
[0]: "github-ee486465-actions-runner-controller-github-webhook-server"
[1]: "github-ee486465-actions-runner-controller-manager"
[2]: "github-ee486465-actions-runner-controller-proxy"
[3]: "github-ee486465-actions-runner-controller-runner-editor"
[4]: "github-ee486465-actions-runner-controller-runner-viewer"
]
ClusterRoleBinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1 : [
[0]: "github-ee486465-actions-runner-controller-github-webhook-server"
[1]: "github-ee486465-actions-runner-controller-manager"
[2]: "github-ee486465-actions-runner-controller-proxy"
]
CustomResourceDefinition.apiextensions.k8s.io/apiextensions.k8s.io/v1 : [
[0]: "horizontalrunnerautoscalers.actions.summerwind.dev"
[1]: "runnerdeployments.actions.summerwind.dev"
[2]: "runnerreplicasets.actions.summerwind.dev"
[3]: "runners.actions.summerwind.dev"
[4]: "runnersets.actions.summerwind.dev"
]
Deployment.apps/apps/v1 : [
[0]: "github/github-ee486465-actions-runner-controller"
[1]: "github/github-ee486465-actions-runner-controller-github-webhook-server"
]
Ingress/ : [
[0]: "github/github-ee486465-actions-runner-controller-github-webhook-server"
]
MutatingWebhookConfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1 : [
[0]: "github-ee486465-actions-runner-controller-mutating-webhook-configuration"
]
Role.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1 : [
[0]: "github/github-ee486465-actions-runner-controller-leader-election"
]
RoleBinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1 : [
[0]: "github/github-ee486465-actions-runner-controller-leader-election"
]
Secret/v1 : [
[0]: "github/controller-manager"
]
Service/v1 : [
[0]: "github/github-ee486465-actions-runner-controller-github-webhook-server"
[1]: "github/github-ee486465-actions-runner-controller-metrics-service"
[2]: "github/github-ee486465-actions-runner-controller-webhook"
]
ServiceAccount/v1 : [
[0]: "github/github-ee486465-actions-runner-controller"
[1]: "github/github-ee486465-actions-runner-controller-github-webhook-server"
]
ValidatingWebhookConfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1: [
[0]: "github-ee486465-actions-runner-controller-validating-webhook-configuration"
]
}
reuseValues : false
skipAwait : false
skipCrds : false
timeout : 300
values : {
authSecret : {
create : true
github_token: [secret]
}
certManagerEnabled : false
githubWebhookServer: {
enabled: true
ingress: {
enabled : true
hosts : [
[0]: {
host : "github.<domain>"
paths: [
[0]: {
path : "/actions-runner-controller-github-webhook-server"
pathType: "Prefix"
}
]
}
]
ingressClassName: "nginx-public"
}
}
replicaCount : 2
}
verify : false
version : "0.20.2"
waitForJobs : false
Expected Behavior
The Ingress resource is to be created in new k8s.helm.v3.Chart.
Actual Behavior
When using new k8s.helm.v3.Chart I don't get the Ingress resource created.
Output of pulumi about
$ pulumi about
CLI
Version 3.38.0
Go Version go1.19
Go Compiler gc
Plugins
NAME VERSION
nodejs unknown
Host
OS darwin
Version 12.4
Arch arm64
This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v18.7.0'
Current Stack: staging
TYPE URN
pulumi:pulumi:Stack urn:pulumi:staging::git::pulumi:pulumi:Stack::git-staging
kubernetes:helm.sh/v3:Chart urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart::github
pulumi:providers:pulumi urn:pulumi:staging::git::pulumi:providers:pulumi::default
pulumi:providers:kubernetes urn:pulumi:staging::git::pulumi:providers:kubernetes::default_3_21_0
pulumi:pulumi:StackReference urn:pulumi:staging::git::pulumi:pulumi:StackReference::kubernetes/staging
kubernetes:core/v1:Namespace urn:pulumi:staging::git::kubernetes:core/v1:Namespace::github
pulumi:providers:kubernetes urn:pulumi:staging::git::pulumi:providers:kubernetes::staging
pulumi:providers:aws urn:pulumi:staging::git::pulumi:providers:aws::default_5_11_0
aws:iam/role:Role urn:pulumi:staging::git::aws:iam/role:Role::github-runners
kubernetes:core/v1:ServiceAccount urn:pulumi:staging::git::kubernetes:core/v1:ServiceAccount::github-organization-runner
aws:iam/rolePolicy:RolePolicy urn:pulumi:staging::git::aws:iam/rolePolicy:RolePolicy::github
kubernetes:actions.summerwind.dev/v1alpha1:RunnerDeployment urn:pulumi:staging::git::kubernetes:actions.summerwind.dev/v1alpha1:RunnerDeployment::github-organization-runner
kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration::github-actions-runner-controller-mutating-webhook-configuration
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::github-actions-runner-controller-manager
kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::github-actions-runner-controller-validating-webhook-configuration
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::github-actions-runner-controller-proxy
kubernetes:core/v1:ServiceAccount urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ServiceAccount::github/github-actions-runner-controller-github-webhook-server
kubernetes:core/v1:ServiceAccount urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ServiceAccount::github/github-actions-runner-controller
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::github-actions-runner-controller-github-webhook-server
kubernetes:core/v1:Service urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::github/github-actions-runner-controller-metrics-service
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::github-actions-runner-controller-runner-editor
kubernetes:core/v1:Service urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::github/github-actions-runner-controller-github-webhook-server
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::github-actions-runner-controller-runner-viewer
kubernetes:core/v1:Service urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::github/github-actions-runner-controller-webhook
kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding::github-actions-runner-controller-github-webhook-server
kubernetes:apps/v1:Deployment urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::github/github-actions-runner-controller-github-webhook-server
kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding::github-actions-runner-controller-proxy
kubernetes:apps/v1:Deployment urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::github/github-actions-runner-controller
kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding::github-actions-runner-controller-manager
kubernetes:rbac.authorization.k8s.io/v1:Role urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:Role::github/github-actions-runner-controller-leader-election
kubernetes:rbac.authorization.k8s.io/v1:RoleBinding urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::github/github-actions-runner-controller-leader-election
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::horizontalrunnerautoscalers.actions.summerwind.dev
kubernetes:actions.summerwind.dev/v1alpha1:HorizontalRunnerAutoscaler urn:pulumi:staging::git::kubernetes:actions.summerwind.dev/v1alpha1:HorizontalRunnerAutoscaler::github-organization-runner-autoscaller
kubernetes:core/v1:Secret urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Secret::github/controller-manager
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::runnerdeployments.actions.summerwind.dev
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::runners.actions.summerwind.dev
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::runnerreplicasets.actions.summerwind.dev
kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition urn:pulumi:staging::git::kubernetes:helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::runnersets.actions.summerwind.dev
Found no pending operations associated with staging
Backend
Name pulumi.com
URL https://app.pulumi.com/igordrnobrega
User igordrnobrega
Organizations igordrnobrega
Pulumi locates its logs in /var/folders/qr/r0vh3p4s4vs3j9dhsx9bx3rh0000gn/T/ by default
Plugins because we use a monorepo to our projects
$ pulumi plugin ls
NAME KIND VERSION SIZE INSTALLED LAST USED
aws resource 5.11.0 410 MB 6 days ago 3 days ago
aws-native resource 0.19.0 84 MB 1 week ago 1 week ago
datadog resource 4.10.0 56 MB 1 week ago 6 days ago
docker resource 3.2.0 41 MB 1 month ago 1 month ago
docker-buildkit resource 0.1.17 23 MB 1 month ago 1 month ago
eks resource 0.41.2 180 MB 6 days ago 6 days ago
github resource 4.16.0 45 MB 23 hours ago 23 hours ago
kubernetes resource 3.21.0 81 MB 1 hour ago 1 hour ago
mysql resource 3.1.0 41 MB 6 months ago 6 months ago
vault resource 5.6.0 48 MB 1 hour ago 9 seconds ago
TOTAL plugin cache size: 5.4 GB
Helm JIC
helm version 180ms Thu Aug 25 16:55:43 2022
version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.19"}
Additional context
IDK if it's important, but with helm command, it also works:
helm upgrade --install --namespace actions-runner-system --create-namespace \
--values values.yaml \
--wait actions-runner-controller actions-runner-controller/actions-runner-controller
Where values.yaml it's just the same options in the mentioned configuration for new k8s.helm.v3.Chart.
Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
Hey guys, any news on this? Because suddenly I see new behavior with what used to work with new k8s.helm.v3.Release is now trying to delete the Ingress resource.
Kubernetes provider updated from 3.21.0 to 3.21.2.
Thanks
Hey guys,
is there any news on this? I've updated it to 3.22.1 and still have the same error.
Thanks
@igordrnobrega sorry for not responding sooner. Are you sure your provider is fully updated to 3.22.1? Existing resources have an embedded link to their provider version. If this is on an existing stack, you will want to update the package dependency (e.g. package.json) to v3.22.1 and then do a single pulumi up.
FWIW it looks like the helm chart template in question uses kubernetes capabilities: https://github.com/actions-runner-controller/actions-runner-controller/blob/v0.20.2/charts/actions-runner-controller/templates/githubwebhook.ingress.yaml#L4 which has been fixed in v3.22.1 thanks to https://github.com/pulumi/pulumi-kubernetes/pull/2155
cc @lblackstone for further follow up if necessary.
