[BUG] Index replacement with dynamic properties

[RomainDubois]

What is the bug?

When mappings allows dynamic fields, a re-apply of an unchanged opensearch_index resource will detect dynamically added fields and will try to replace the index.

How can one reproduce the bug?

First create and index:

resource "opensearch_index" "dynamic_index" {
  name               = "dynamic-index"
  number_of_shards   = "1"
  number_of_replicas = "1"
  mappings           = "{}"
}

Then add a document with a field:

PUT /dynamic-index/_doc/my-id
{
  "key": "value"
}

Re-apply the initial resource.

What is the expected behavior?

No change should be detected.

What is your host/environment?

Opensearch provider 2.2.1

Do you have any screenshots?

Do you have any additional context?

Versions before 2.2.1 of the provider do not detect changes. I think it is related to this PR: https://github.com/opensearch-project/terraform-provider-opensearch/pull/145

[sergiojoker11]

This is arguably not a breaking version.

[fpaterour-ippon]

Hi same issue here. We use a separate null_resource to deal with mapping changes since the opensearch_index doesn't support mapping updates without index recreation. So we create the index without the mapping parameter and now after the refresh there is a drift with the state and Terraform tries to delete the mapping, systematically triggering the index recreation that we were trying to avoid. Changing version = "~> 2.2" to version = "2.2.0" solved the issue.

[RomainDubois]

As a workaround, we are trying to ignore mapping changes with:

  lifecycle {
    ignore_changes = [mappings]
  }

Still under test though.

[sergiojoker11]

Above has worked for us.

[rblcoder]

When there are no documents present in the index, updating the index mapping through terraform code works as expected. The index mapping is updated successfully.

Using the following terraform code, when I first create the index

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.2.1"
    }
  }
}

provider "opensearch" {
  url = "https://127.0.0.1:9200"
  healthcheck       = "false"

  username          = "admin"
  password          = "myStrongPassword123@456"
  insecure          = "true"

}

resource "opensearch_index" "test" {
  name               = "terraform-test"
  mappings           = "{}"
}

The index mapping

GET /terraform-test/_mapping

gives

{
  "terraform-test": {
    "mappings": {}
  }
}

Then, I insert a document using REST API

PUT /terraform-test/_doc/1
{
  "name" : "John Doe"
}
 

Now the mapping is

{
  "terraform-test": {
    "mappings": {
      "properties": {
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

Now when I run terraform plan

$ terraform plan
opensearch_index.test: Refreshing state... [id=terraform-test]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # opensearch_index.test must be replaced
-/+ resource "opensearch_index" "test" {
      ~ id                 = "terraform-test" -> (known after apply)
      ~ mappings           = jsonencode(
          ~ {
              - properties = {
                  - name = {
                      - fields = {
                          - keyword = {
                              - ignore_above = 256
                              - type         = "keyword"
                            }
                        }
                      - type   = "text"
                    }
                }
            } # forces replacement
        )
        name               = "terraform-test"
      ~ number_of_replicas = "1" -> (known after apply)
      ~ number_of_shards   = "1" -> (known after apply)
      + rollover_alias     = (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these
actions if you run "terraform apply" now.

Mappings has

ForceNew:     true,

causing the index to be recreated on re-apply. https://github.com/opensearch-project/terraform-provider-opensearch/blob/174c272a967fbb913abbe3e5baba4b677cc6666b/provider/resource_opensearch_index.go#L368-L373

OpenSearch documentation on Create or update mappings https://opensearch.org/docs/latest/api-reference/index-apis/put-mapping/

Now if I remove "ForceNew: true," run the terraform provider in debug mode and run terraform plan, the index will not be recreated

 $ terraform plan
opensearch_index.test: Refreshing state... [id=terraform-test]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # opensearch_index.test will be updated in-place
  ~ resource "opensearch_index" "test" {
        id                 = "terraform-test"
      ~ mappings           = jsonencode(
          ~ {
              - properties = {
                  - name = {
                      - fields = {
                          - keyword = {
                              - ignore_above = 256
                              - type         = "keyword"
                            }
                        }
                      - type   = "text"
                    }
                }
            }
        )
        name               = "terraform-test"
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these
actions if you run "terraform apply" now.

Now the mapping still is

{
  "terraform-test": {
    "mappings": {
      "properties": {
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

If documents are present in the index, we need to consult the documentation https://opensearch.org/docs/latest/api-reference/index-apis/put-mapping/

PR https://github.com/opensearch-project/terraform-provider-opensearch/pull/145 was made to fix #135 #71

[rblcoder]

aws_dynamodb_table resource recommends using lifecycle ignore_changes for some arguments in certain situations https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dynamodb_table

[ms-semarchy]

In my case the bug also happens when static mappings are defined. Provider version: 2.3.1