terraform-provider-ec
terraform-provider-ec copied to clipboard
elastic
Race condition on creating Deployment with custom config that relies on keystore secret
In scenario when you specify custom elasticsearch.yaml settings using elasticsearch.config.user_settings_yaml and settings provided there requires you to use keystore for storing secure parameters you end in race condition when creating new deployment / editing existing.
Readiness Checklist
- [x] I am running the latest version (yes 3.0)
- [x] I checked the documentation and found no answer
- [x] I checked to make sure that this issue has not already been filed
- [x] I am reporting the issue to the correct repository (for multi-repository projects)
Expected Behavior
Apply should work without failing.
Current Behavior
Example of configuration documentation: https://www.elastic.co/guide/en/cloud/current/ec-securing-clusters-oidc-op.html#ec-securing-oidc-okta (steps 2 & 3)
What is happening:
- Deployment is being created/updated with new
elasticsearch.yamlsettings - Keystore secret depends on deployment so it is waiting for 1. to finish
- Elasticsearch cannot be restarted in step 1. which makes apply either hang (when creating deployment from scratch) or immediately fail (when applying to existing deployment).
## Terraform definition
resource "ec_deployment" "race-condition-cluster" {
name = "race-condition-cluster"
region = "eu-west-1"
version = "7.16.3"
deployment_template_id = "aws-observability"
elasticsearch {
topology {
id = "hot_content"
size = "1g"
zone_count = 1
}
config {
user_settings_yaml = <<EOF
xpack.security.authc.realms.oidc.oidc1:
order: 2
rp.client_id: client_id
rp.response_type: "code"
rp.requested_scopes: ["openid", "email"]
rp.redirect_uri: "https://kibana.example.com/api/security/oidc/callback"
op.issuer: "https://sample.okta.com"
op.authorization_endpoint: "https://sample.okta.com/oauth2/v1/authorize"
op.token_endpoint: "https://sample.okta.com/oauth2/v1/token"
op.userinfo_endpoint: "https://sample.okta.com/oauth2/v1/userinfo"
op.endsession_endpoint: "https://sample.okta.com/oauth2/v1/logout"
op.jwkset_path: "https://sample.okta.com/oauth2/v1/keys"
claims.principal: email
claim_patterns.principal: "^([^@]+)@sample\\.com$"
EOF
}
}
kibana {
topology {
size = "1g"
zone_count = 1
}
config {
user_settings_yaml = <<EOF
xpack.security.authc.providers:
oidc.oidc1:
order: 0
realm: oidc1
description: "Log in with Okta"
EOF
}
}
}
resource "ec_deployment_elasticsearch_keystore" "okta_secret" {
deployment_id = ec_deployment.race-condition-cluster.id
setting_name = "xpack.security.authc.realms.oidc.oidc1.rp.client_secret"
value = "my_super_secret"
}
Steps to Reproduce
Apply sample terraform config provided above
Context
We are able to deal with it by simply temporarily commenting out custom settings and doing two applies to fully apply this feature. We would love to have it without manual operations to be done
Possible Solution
Allow to configure custom elasticsearch.yml entries via separate TF resource
Your Environment
- Version used: 0.3.0
- Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: EC SaaS
- Environment name and version (e.g. Go 1.9): Terraform 1.1.4
We are facing the same issue. Would love to see this fixed. Besides this we also saw other use cases when a dedicated resource for user settings would make sense - in the original example, it might be the redirect_url or anything else that is known only after the Elasticsearch/Kibana instance is created.
+1 for the suggestion to have a separate terraform resource to configure a deployment's settings (both elastic and kibana).
+1. This race condition doesn't allow me to set up SSO on cluster creation.
Same here. The race condition is preventing us from moving to the terraform-provider-ec. We were looking to migrate from using local-exec to execute the API call for the deployment and SSO config and use the provider instead.
It would be nice if this could be addressed, our automation is currently slowed down by this.
Any fix for this yet ?
The newer version of ec provider (0.6) helps with resolving circular dependency of kibana URL and elasticsearch.yaml file, but however the deployments are getting failed due to dependency on keystore value.
Below error was noticed -
'''
[instance-0000000001] fatal exception while booting Elasticsearch java.lang.IllegalStateException: security initialization failed at org.elasticsearch.xpack.security.Security.createComponents(Security.java:578) ~[?:?] at org.elasticsearch.node.Node.lambda$new$16(Node.java:721) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252) ~[elasticsearch-8.6.2.jar:?] at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?] at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?] at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?] at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?] at org.elasticsearch.node.Node.
@saimantr this is a limitation on the Elastic Cloud API at the moment. There's a request there to support setting keystore values during deployment creation, however there's no plans to work on that in the near future.
@saimantr this is a limitation on the Elastic Cloud API at the moment. There's a request there to support setting keystore values during deployment creation, however there's no plans to work on that in the near future.
Thanks for the revert @tobio. So, my use case is to provision an elasticsearch cluster with SSO integration in one go using terraform. IS there any alternative way to achieve this or any thoughts ?
Another ignored ticket :-/ Holding up progress on https://github.com/elastic/terraform-provider-ec/issues/656
@tommed , we're working on it. I'm sorry for the delay but we have to change both the provider and ECE API to properly handle that use case.
@dimuon Apologies for the comment, but we are finding that a lot of tickets relating to terraform (especially those in hashicorp's own repos) are being left to stagnate, even if the issue is a simple documentation error. It gets quite frustrating when you've convinced a team to move all DSC/IaC work into Terraform and then are stuck because of tickets raised months ago. Best of luck with this one!! Once we have a fix, we will be able to automate our entire service stack via Terraform Cloud - which will be amazing to have working again!
Should be addressed by https://github.com/elastic/terraform-provider-ec/pull/674
@tommed , the needed change in backend was deployed last week so we merged the corresponding PR for the provider and released version 0.9.0 that addresses the issue.
Any fix for this yet ? The newer version of ec provider (0.6) helps with resolving circular dependency of kibana URL and elasticsearch.yaml file, but however the deployments are getting failed due to dependency on keystore value. Below error was noticed - ''' [instance-0000000001] fatal exception while booting Elasticsearch java.lang.IllegalStateException: security initialization failed at org.elasticsearch.xpack.security.Security.createComponents(Security.java:578) ~[?:?] at org.elasticsearch.node.Node.lambda$new$16(Node.java:721) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252) ~[elasticsearch-8.6.2.jar:?] at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?] at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?] at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?] at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?] at org.elasticsearch.node.Node.(Node.java:736) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.node.Node.(Node.java:322) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.6.2.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.6.2.jar:?] Caused by: org.elasticsearch.common.settings.SettingsException: The configuration setting [xpack.security.authc.realms.oidc.oidc1.rp.client_secret] is required at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectRealm.buildRelyingPartyConfiguration(OpenIdConnectRealm.java:256) ~[?:?] at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectRealm.(OpenIdConnectRealm.java:98) ~[?:?] at org.elasticsearch.xpack.security.authc.InternalRealms.lambda$getFactories$7(InternalRealms.java:169) ~[?:?] at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:288) ~[?:?] at org.elasticsearch.xpack.security.authc.Realms.(Realms.java:109) ~[?:?] at org.elasticsearch.xpack.security.Security.createComponents(Security.java:686) ~[?:?] at org.elasticsearch.xpack.security.Security.createComponents(Security.java:566) ~[?:?] ... 17 more '''
hey @dimuon @tobio ; I can still see the same logs and error while trying to configure OIDC for SSO login while creating the cluster. Any thoughts around this ?
@saimantr , do you use 0.9.0 against Elastic cloud or ECE? Can you please share your TF config?
@dimuon : We are using Elasticsearch Service for this and yes set the version to 0.9.0. Terraform version is 1.4.6
terraform {
required_version = ">= 0.13.1"
required_providers {
random = {
source = "hashicorp/random"
version = "3.5.1"
}
ec = {
source = "elastic/ec"
version = "0.9.0"
}
elasticstack = {
source = "elastic/elasticstack"
version = "0.5.0"
}
}
}
provider "ec" {
apikey = var.ec_api_key
}
resource "random_uuid" "uuid" {}
#Resource block actually responsible for creating elastic deployment.
resource "ec_deployment" "ecaas_deployment" {
name = “test123”
region = var.region
version = var.ec_version. #ec version used is 8.9.2
deployment_template_id = var.deployment_template_id. #aws storage optimized
elasticsearch = {
autoscale = var.autoscale. #And for this exp; I've set autoscale to "true"
config = {
user_settings_yaml = templatefile("./ec_user_settings.yaml.tftpl”)
}
cold = {
autoscaling = var.autoscale ? {
max_size = "0g"
max_size_resource = "memory"
} : {}
}
frozen = {
size = var.autoscale ? var.frozen_size_def : var.frozen_size
zone_count = var.frozen_count
autoscaling = var.autoscale ? {
max_size = var.frozen_size
max_size_resource = "memory"
} : {}
}
hot = {
size = var.autoscale ? var.hot_size_def : var.hot_size
zone_count = var.hot_count
autoscaling = var.autoscale ? {
max_size = var.hot_size
max_size_resource = "memory"
} : {}
}
ml = {
autoscaling = var.autoscale ? {
max_size = "0g"
max_size_resource = "memory"
} : {}
}
warm = {
autoscaling = var.autoscale ? {
max_size = "0g"
max_size_resource = "memory"
} : {}
}
}
kibana = {
topology = {
size = var.kibana_size
zone_count = var.kibana_count
}
config = {
user_settings_yaml = file("./kibana_user_settings.yaml")
}
}
integrations_server = {
topology = {
size = var.integrations_server_size
zone_count = var.integrations_server_count
}
}
}
resource "ec_deployment_elasticsearch_keystore" "client_secret" {
deployment_id = ec_deployment.ecaas_deployment.id
setting_name = "xpack.security.authc.realms.oidc.oidc1.rp.client_secret"
value = var.ec_keystore_client_secret
}
@saimantr , you need to move client_secret definition to new attribute keystore_contents of ec_deployment. Please refer to the example for more details.