[Bug] Data stream lifecycles on Elastic Cloud serverless can't be deleted

[respondersGY]

Describe the bug Deleting the resource resource elasticstack_elasticsearch_data_stream_lifecycle is not possible with the Terraform provider. Nonethless, it is possible in the Kibana UI that uses the following API endpoint: api/index_management/delete_data_streams.

To Reproduce

resource "elasticstack_elasticsearch_index_template" "main" {
  name     = "test"
  priority = 500

  index_patterns = ["test*"]

  data_stream {}

  template { }
}

resource "elasticstack_elasticsearch_data_stream" "main" {
  name = "test"

  depends_on = [
    elasticstack_elasticsearch_index_template.main
  ]
}

resource "elasticstack_elasticsearch_data_stream_lifecycle" "main" {
  name               = "test"
  data_retention = null

  depends_on = [
    elasticstack_elasticsearch_data_stream.main,
  ]

The error:

Failed with: {"error":{"root_cause":[{"type":"api_not_available_exception","reason":"Request for uri
│ [/_data_stream/test/_lifecycle?expand_wildcards=open] with method [DELETE] exists but is not available when running
│ in serverless mode"}],"type":"api_not_available_exception","reason":"Request for uri
│ [/_data_stream/test/_lifecycle?expand_wildcards=open] with method [DELETE] exists but is not available when running
│ in serverless mode"},"status":410}

Expected behavior The resource elasticstack_elasticsearch_data_stream_lifecycle is correctly deleted. In the same manner that this can be done in the Kibana UI (api/index_management/delete_data_streams).

If this is not possible due to API limitations, it should be documented that use of this resource requires a manual removal from the Terraform state when destroying the infrastructure (tf state rmelasticstack_elasticsearch_data_stream_lifecycle.main)?

[tobio]

Deleting the resource resource elasticstack_elasticsearch_data_stream_lifecycle is not possible with the Terraform provider. Nonethless, it is possible in the Kibana UI that uses the following API endpoint: api/index_management/delete_data_streams.

For clarity, these are two different operations. Deleting the lifecycle resource only removes the lifecycle, leaving the underlying data stream unmanaged. Deleting the actual data stream from Kibana is removing the underlying data.

There's still an issue to be resolved here.

[respondersGY]

Workaround that requires two executions of terraform apply.

resource "null_resource" "remove_data_stream" {
  triggers = {
    kibana_https_url                              = ec_elasticsearch_project.main.endpoints.kibana
    kibana_username                             = ec_elasticsearch_project.main.credentials.username
    kibana_password                             = sensitive(ec_elasticsearch_project.main.credentials.password)
    elasticsearch_data_stream_name = "test"
  }


  provisioner "local-exec" {
    when         = destroy
    quiet         = true
    on_failure = fail
    command  = <<EOT
      set -e 
      curl \
        ${self.triggers.kibana_https_url}/api/index_management/delete_data_streams \
        -u '${self.triggers.kibana_username}:${self.triggers.kibana_password}' \
        -H 'content-type: application/json' \
        -H 'kbn-xsrf: true' \
        -H 'x-elastic-internal-origin: Kibana' \
        --data-raw '{"dataStreams":["${self.triggers.elasticsearch_data_stream_name}"]}'
    EOT
  }

  depends_on = [
    elasticstack_elasticsearch_data_stream.main,
    elasticstack_elasticsearch_data_stream_lifecycle.main,
  ]
}