terraform-aws-eks-jx
terraform-aws-eks-jx copied to clipboard
Vault instance failing to create
Summary
I am having an issue where the vault instance will not create. The operator is up and running, but when the helm provider tries to create the vault instance it fails
Steps to reproduce the behavior
- use an existing cluster
- tf init, plan, apply
Expected behavior
The vault instance is created
Actual behavior
Warning: Helm release "vault-instance" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
with module.vault.helm_release.vault-instance[0],
on modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
11: resource "helm_release" "vault-instance" {
Error: Vault.vault.banzaicloud.com "vault" is invalid: spec.ingress.annotations: Invalid value: "null": spec.ingress.annotations in body must be of type object: "null"
with module.vault.helm_release.vault-instance[0],
on modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
11: resource "helm_release" "vault-instance" {
Terraform version
The output of terraform version
is:
Terraform v1.1.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.75.1
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.5.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.9.0
+ provider registry.terraform.io/hashicorp/local v2.2.2
+ provider registry.terraform.io/hashicorp/null v3.1.1
+ provider registry.terraform.io/hashicorp/random v3.1.2
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1
Module version
master
Operating system
Linux inside of the container
I downgraded to using v1.18.11 with a lower version of Terraform and I am getting the same result:
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [40s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [50s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [1m0s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Creation complete after 1m7s [id=nginx-ingress]
Warning: Helm release "vault-instance" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
on .terraform/modules/eks-jx/modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
11: resource "helm_release" "vault-instance" {
Error: Vault.vault.banzaicloud.com "vault" is invalid: spec.ingress.annotations: Invalid value: "null": spec.ingress.annotations in body must be of type object: "null"
on .terraform/modules/eks-jx/modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
11: resource "helm_release" "vault-instance" {
Here is my main.tf
module "eks-jx" {
source = "jenkins-x/eks-jx/aws"
region = var.region
use_vault = var.use_vault
use_asm = var.use_asm
cluster_name = var.cluster_name
is_jx2 = var.is_jx2
create_eks = var.create_eks
create_vpc = var.create_vpc
create_nginx = var.create_nginx
jx_git_url = var.jx_git_url
apex_domain = var.apex_domain
tls_email = var.tls_email
use_kms_s3 = var.use_kms_s3
registry = var.registry
nginx_chart_version = var.nginx_chart_version
cluster_version = var.cluster_version
enable_backup = var.enable_backup
jx_bot_username = var.jx_bot_username
jx_bot_token = var.jx_bot_token
enable_external_dns = var.enable_external_dns
jx_git_operator_values = var.jx_git_operator_values
production_letsencrypt = var.production_letsencrypt
}
My vars file
cluster_name="foo"
cluster_version="1.19"
region="us-west-2"
create_nginx="true"
jx_git_url="https://gitlab.com"
jx_bot_username="chrislovecnm"
enable_backup="false"
apex_domain="api-jx3.foo.com"
tls_email="[email protected]"
enable_external_dns=true
production_letsencrypt="true"
use_kms_s3="true"
registry="foo.dkr.ecr.us-east-2.amazonaws.com"
jx_git_operator_values = [
"gitKind: gitlab",
"environmentGitOwner: foo"
]
nginx_chart_version="4.0.19"
create_eks=false
create_vpc=false
use_vault=true
cluster_in_private_subnet=true
I wonder if it's because you are using 1.19 version of kubernetes, is it possible to use 1.21 and see if it works?
Yes I can do that
Yep, it works with 1.21. Do you have a support matrix listed?
Yep, it works with 1.21. Do you have a support matrix listed?
Atm we dont have one (we support 1.20+ afaict), but we dont support 1.22 yet (but very close to supporting it - a few helm charts need an upgrade)
We will be adding a few e2e tests to help us create a matrix soon.
I can close this, but should we have a support matrix first. I think there was a change to the crd api between 1.19 and 1.21 … if I recall
I can close this, but should we have a support matrix first
Agreed, I am fixing our internal infrastructure this week, and then we plan to add support for 1.22. Once that work is done, I am going to focus on adding kind tests to run tests on different cluster versions on every PR to jx3-version repo.