Output are not always correctly populated

[kuisathaverat]

Readiness Checklist

• [x] I am running the latest version
• [x] I checked the documentation and found no answer
• [x] I checked to make sure that this issue has not already been filed
• [x] I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

The defined outputs are set with the correct values.

Current Behavior

Not all output with data related to Integrations Server, APM ,and Kibana are correctly populated.

## Terraform definition

Deploy only Elasticsearch main.tf(1)

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
  }

  kibana {
    topology {
      size = "0g"
    }
  }

  integrations_server {
    topology {
      zone_count = 2
      size = "0g"
    }
  }
}

Update with Kibana and Integrations server main.tf(2)

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
  }

  kibana {
    topology {
      size = "2g"
    }
  }

  integrations_server {
    topology {
      zone_count = 2
      size = "2g"
    }
  }
}

Outputs configured at output.tf file

output "main_kibana_https_endpoint" {
  value = ec_deployment.main.kibana[0].https_endpoint
}

output "main_elasticsearch_https_endpoint" {
  value = ec_deployment.main.elasticsearch[0].https_endpoint
}

output "main_apm_https_endpoint" {
  value = ec_deployment.main.integrations_server[0].https_endpoint
}

Steps to Reproduce

1. copy the main.tf(1) and the output.tf files in a folder
2. execute terraform apply, the deploy will have only Elasticsearch
3. add resources to Kibana and integrations_server see main.tf(2)
4. execute terraform apply you will see that some outputs are not populated with the correct values

Outputs:

main_apm_https_endpoint = ""
main_elasticsearch_https_endpoint = "https://FOO.us-west2.gcp.elastic-cloud.com:9243"
main_kibana_https_endpoint = ""

5. execute again terraform apply no changes will apply but the output will be populated correctly

Context

In order to restore data we deploy only Elasticsearch, we restore the data and then we start the rest of the Elastic Stack, we use outputs to configure other processes/services that depend on those details to connect to the cluster. To not have correct outputs break out process.

Possible Solution

We've changed to use the terraform status file due to the outputs do not contain the correct details in some cases.

Your Environment

• Version used: 0.4.0
• Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: ESS
• Environment name and version (e.g. Go 1.9):
• Server type and version:
• Operating System and version: macOS
• Link to your project:

[dimuon]

I think this is a limitation of our implementation.

The output variable isn’t reevaluated by Terraform because its value already known (it’s empty string in our case) even despite the fact that the state contains a new value.

The general solution in such cases is to mark the argument kibana.0.https_endpoint as Computed if kibana.0.topology.size attribute is changed. This logic has to be implemented in CustomizeDiff field of ec_deployment resource but Terraform cannot mark nested fields as Computed - it fails with SetNewComputed: invalid key: kibana.0.https_endpoint.

It is possible that the limitation can be removed after the transition to the plugin platform, but this is for further study.

@tobio , what do you think?

[tobio]

@dimuon AFAICT that's correct.

From the docs we also have this which seems to support the conclusion:

NOTE: CustomizeDiff does not currently support computed/"known after apply" values from other resource attributes.

[andrewnazarov]

Faced the same issue after moving from apm {} to integrations_server {}. The output wasn't populated and I had to do terraform apply twice.

[tobio]

Fixed in 0.6.0