Investigate for Kibana Alert and OpenSearch Alert

[timckt]

Background

When we run terraform plan and terraform apply for OpenSearch Alert at account and Kibana alert at global-resources, there will always be change for it even we do not update the related terraform code. Fix it to prevent confusion.

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] readme has been updated
• [ ] user docs have been updated
• [ ] another team member has reviewed
• [ ] smoke tests are green
• [ ] prepare demo for the team

Reference

How to write good user stories

[timckt]

There are 6 resources at app-opensearch-alerts-cp.tf file, 1 is the slack channel configuration and 5 are alerts. Those 6 files will always be "changed" when we run terraform plan.

• For the slack channel configuration, we need to remove it and then recreate it with "config_id" inside body, then it will not be changed every time.

resource "opensearch_channel_configuration" "cloud_platform_alerts" {
  provider = opensearch.app_logs
  body     = <<EOF
{
  "config_id": "slack-notify",
  "config": {
    "name": "cloud-platform-alerts",
    "description" : "cloud-platform-alerts",
    "config_type" : "slack",
    "is_enabled" : true,
    "slack": {
      "url": "${jsondecode(data.aws_secretsmanager_secret_version.slack_webhook_url.secret_string)["url"]}"
    }
  }
}
EOF
}

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

[timckt]

• For the 5 alerts resource, we are now creating those by below code.

resource "opensearch_monitor" "movies_last_hour" {
  body = <<EOF
{
  "name": "test-monitor",
  "type": "monitor",
  "enabled": true,
 ......
EOF
}

• Changed the method to use templatefile with new tpl file, but still get change every time in terraform plan.

resource "opensearch_monitor" "duplicate_grafana_uid_monitor" {
  provider = opensearch.app_logs
  body = templatefile("${path.module}/resources/opensearch/duplicate_grafana_uid_monitor.tpl", {
......
  })

}

• It seems like a bug for the provider. Can see some other report from https://github.com/opensearch-project/terraform-provider-opensearch/issues/100.

[timckt]

Tried with different approach but still not working for OpenSearch Alert, more details can be found in https://docs.google.com/document/d/1nnVeHrLCFZUbVCDR8Cp-KQTUv03rrD830J6hzqeNwm4/edit?usp=sharing

[timckt]

opensearch_channel_configuration and opensearch_monitor now will not make change in terraform plan every time now

But elasticsearch_opensearch_role.all_org_members still get a change in terraform plan

https://concourse.cloud-platform.service.justice.gov.uk/teams/main/pipelines/infrastructure-account/jobs/terraform-plan/builds/83.1