terraform-provider-rancher2
terraform-provider-rancher2 copied to clipboard
rancher
Add admission_configuration configuration
Fixes https://github.com/rancher/terraform-provider-rancher2/issues/908
I'm rather new to write Golang and using the terraform SDK. I'm open to any help in improving this code.
Unfortunately as specified here: https://github.com/rancher/rancher/issues/37455 AdmissionConfiguration type coming from K8s is erased and is mapped to map[string]interface{} this forces me to do a lot of casting to make it work.
With this fix with the following terraform code:
admission_configuration {
api_version = "apiserver.config.k8s.io/v1"
kind = "AdmissionConfiguration"
plugins {
name = "PodSecurity"
path = ""
configuration = file("files/security.yml")
}
plugins {
name = "EventRateLimit"
path = ""
configuration = file("files/ratelimit.yml")
}
}
We obtain the following code in the terraform state
"admission_configuration": [
{
"api_version": "apiserver.config.k8s.io/v1",
"kind": "AdmissionConfiguration",
"plugins": [
{
"configuration": "apiVersion: pod-security.admission.config.k8s.io/v1alpha1\ndefaults:\n audit: restricted\n audit-version: latest\n enforce: restricted\n enforce-version: latest\n warn: restricted\n warn-version: latest\nexemptions:\n namespaces: []\n runtimeClasses: []\n usernames: []\nkind: PodSecurityConfiguration\n",
"name": "PodSecurity",
"path": ""
},
{
"configuration": "apiVersion: eventratelimit.admission.k8s.io/v1alpha1\nkind: Configuration\nlimits:\n- burst: 20000\n qps: 5000\n type: Server\n",
"name": "EventRateLimit",
"path": ""
}
]
}
]
@annablender Hey! Let me know if I can improve something in the PR or if you need something else before someone can review it! :)
Hey @annablender Thanks for giving this PR a look. It's been a long time but if I remember correctly I basically used an AdmissionConfiguration file and reverse engineered from there. I'm using my PR in production for the last few months without any issue.
Here is my main.tf with my compile plugin:
terraform {
required_providers {
rancher2 = {
# source = "rancher/rancher2"
# Waiting on https://github.com/rancher/terraform-provider-rancher2/pull/909
source = "terraform.liip.ch/rancher/rancher2"
version = "1.23.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.9.0"
}
}
}
provider "rancher2" {
api_url = "https://rancher.example.com"
}
And here is my cluster definition:
resource "rancher2_cluster" "prod" {
name = "XXXX"
description = "XXX Cluster"
enable_cluster_alerting = false
enable_cluster_monitoring = false
enable_network_policy = true
rke_config {
kubernetes_version = "v1.23.7-rancher1-1"
services {
kube_api {
extra_args = {
enable-admission-plugins = "PodNodeSelector,NodeRestriction"
feature-gates = "PodSecurity=true"
}
pod_security_policy = false
secrets_encryption_config {
enabled = true
}
audit_log {
enabled = true
configuration {
max_age = 30
max_backup = 10
max_size = 100
}
}
event_rate_limit {
enabled = true
}
admission_configuration {
api_version = "apiserver.config.k8s.io/v1"
kind = "AdmissionConfiguration"
plugins {
name = "PodSecurity"
path = ""
configuration = file("../files/security.yml")
}
plugins {
name = "EventRateLimit"
path = ""
configuration = file("../files/ratelimit.yml")
}
}
}
}
}
}
Here is an example for the security.yml file:
apiVersion: pod-security.admission.config.k8s.io/v1alpha1
kind: PodSecurityConfiguration
defaults:
enforce: restricted
enforce-version: latest
audit: restricted
audit-version: latest
warn: restricted
warn-version: latest
exemptions:
usernames: []
runtimeClasses: []
namespaces: []
Let me know if you need anything else
@HarrisonWAffel: I tried adding unit tests. This made me modify my code a bit especially the return types and the parameters of my functions.
However now I get the following error:
--- FAIL: TestFlattenClusterTemplate (0.00s)
panic: can't convert []map[string]interface {}{map[string]interface {}{"configuration":"apiVersion: eventratelimit.admission.k8s.io/v1alpha1\nkind: Configuration\nlimits:\n- {}\n", "name":"EventRateLimit", "path":""}} to cty.Value [recovered]
panic: can't convert []map[string]interface {}{map[string]interface {}{"configuration":"apiVersion: eventratelimit.admission.k8s.io/v1alpha1\nkind: Configuration\nlimits:\n- {}\n", "name":"EventRateLimit", "path":""}} to cty.Value
As I'm clearly still a newbie in golang I'm a bit lost on what could cause this issue. Any help would be appreciated.
Testing it in my repo I get:
panic: interface conversion: interface {} is map[string]interface {}, not []map[string]interface {}
goroutine 146 [running]:
github.com/rancher/terraform-provider-rancher2/rancher2.expandClusterRKEConfigServicesKubeAPIAdmissionConfigurationPlugins({0xc000a48760?, 0xc001215b90?, 0x1f24afb?})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/structure_cluster_rke_config_services_kube_api.go:268 +0x4b1
github.com/rancher/terraform-provider-rancher2/rancher2.expandClusterRKEConfigServicesKubeAPIAdmissionConfiguration({0xc000fc6720, 0x1, 0x1f4372a?})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/structure_cluster_rke_config_services_kube_api.go:302 +0x20a
github.com/rancher/terraform-provider-rancher2/rancher2.expandClusterRKEConfigServicesKubeAPI({0xc000fc66e0, 0x1, 0x1f26153?})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/structure_cluster_rke_config_services_kube_api.go:365 +0xdf
github.com/rancher/terraform-provider-rancher2/rancher2.expandClusterRKEConfigServices({0xc000fc6600, 0x1, 0x1f2668b?})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/structure_cluster_rke_config_services.go:94 +0x165
github.com/rancher/terraform-provider-rancher2/rancher2.expandClusterRKEConfig({0xc000fc65a0, 0x1, 0x1f22c1b?}, {0xc001502210, 0x5})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/structure_cluster_rke_config.go:319 +0xbe8
github.com/rancher/terraform-provider-rancher2/rancher2.resourceRancher2ClusterUpdate(0xc0003376c0, {0x1f1a4c0?, 0xc000478f00})
/home/jocelyn/Liip/terraform-provider-rancher2/rancher2/resource_rancher2_cluster.go:349 +0x171e
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).Apply(0xc000ccfa70, 0xc0006f4b90, 0xc000223ee0, {0x1f1a4c0, 0xc000478f00})
/home/jocelyn/go/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:316 +0x3e3
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Apply(0xc0002c4580, 0xc00108b978, 0x1f32506?, 0xf?)
/home/jocelyn/go/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/provider.go:294 +0x70
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ApplyResourceChange(0xc000c095b0, {0xc0005ec850?, 0x4bc1e6?}, 0xc0005ec850)
/home/jocelyn/go/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:885 +0x7c5
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x1e5a9a0?, 0xc000c095b0}, {0x23752e8, 0xc001232e40}, 0xc0005ec7e0, 0x0)
/home/jocelyn/go/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3305 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000003c0, {0x237c738, 0xc000f9a1a0}, 0xc0012399e0, 0xc000f90210, 0x3355800, 0x0)
/home/jocelyn/go/pkg/mod/google.golang.org/[email protected]/server.go:1295 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc0000003c0, {0x237c738, 0xc000f9a1a0}, 0xc0012399e0, 0x0)
/home/jocelyn/go/pkg/mod/google.golang.org/[email protected]/server.go:1636 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/home/jocelyn/go/pkg/mod/google.golang.org/[email protected]/server.go:932 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/jocelyn/go/pkg/mod/google.golang.org/[email protected]/server.go:930 +0x28a
@jocelynthode The root cause of the issue seems to be the use of []map[string]interface{} in some parts of this PR, specifically how the plugins are being handled for the admission configuration. Looking at the terraform SDK's source code, The call to schema.TestResourceDataRaw done in the TestFlattenClusterTemplate test (rancher2/structure_cluster_template_test.go) will eventually hit the following type switch
// terraform-plugin-sdk/internal/configs/hcl2shim/values.go:196
switch tv := v.(type) {
case bool:
return cty.BoolVal(tv)
case string:
return cty.StringVal(tv)
case int:
return cty.NumberIntVal(int64(tv))
case float64:
return cty.NumberFloatVal(tv)
case []interface{}:
vals := make([]cty.Value, len(tv))
for i, ev := range tv {
vals[i] = HCL2ValueFromConfigValue(ev)
}
return cty.TupleVal(vals)
case map[string]interface{}:
vals := map[string]cty.Value{}
for k, ev := range tv {
vals[k] = HCL2ValueFromConfigValue(ev)
}
return cty.ObjectVal(vals)
default:
// HCL/HIL should never generate anything that isn't caught by
// the above, so if we get here something has gone very wrong.
panic(fmt.Errorf("can't convert %#v to cty.Value", v))
}
As we can see the type []map[string]interface{} is not present, which leads to the panic you're seeing. This panic can be avoided if you change the type from []map[string]interface{} to []interface{}{} and adding a map[string]interface{} as an element. This would follow a similar pattern already used in the unit tests (see he testClusterTemplateRevisionsScheduledClusterScanInterface variable located in rancher2/structure_cluster_template_test.go:122).
In go, []interface{}{map[string]interface{}} is effectively equivalent to []map[string]interface{}, with the only difference being the inability to determine the type of the array elements. From what I can see your new schema looks correct, it's just how it's being flattened and expanded that is causing the issue. lmk if this makes sense or if you need further help
@HarrisonWAffel: Thanks! Your explanation was clear and I was able to fix my issue! I should have thought about digging into the provider code, my bad!