Unexpected warnings when no changes made to alert channels

[danielgblanco]

trafficstars

Terraform Version

terraform 1.1.7
newrelic/newrelic 2.41.2

Affected Resource

• newrelic_alert_channel

Terraform Configuration

Terraform config

provider "newrelic" {
  account_id = "xxxxx"
  api_key = "xxxxx"
  region = "US"
}

Notification channel:

resource "newrelic_alert_channel" "vo-my-routingkey" {
  name = "my-routingkey"
  type = "victorops"
  config {
    key       = "xxxxxx"
    route_key = "my-routingkey"
  }
}

resource "newrelic_alert_channel" "slack-my-channel" {
  name = "my-channe"
  type = "slack"
  config {
    url = "xxxxxx"
    channel = "my-channel"
  }
}

Actual Behavior

This started happening a few weeks ago (difficult to know exactly when). Running a plan with no changes to the notification channels above (inside or outside Terraform because we don't allow UI access to notification channels) results in the following warning:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # newrelic_alert_channel.slack-my-channel has changed
  ~ resource "newrelic_alert_channel" "slack-my-channel" {
        id   = "xxxxxx"
        name = "my-channel"
        # (1 unchanged attribute hidden)

      ~ config {
          + headers = (sensitive value)
          + payload = (sensitive value)
            # (2 unchanged attributes hidden)
        }
    }

  # newrelic_alert_channel.vo-my-routingkey has changed
  ~ resource "newrelic_alert_channel" "vo-my-routingkey" {
        id   = "xxxxxx"
        name = "my-routingkey"
        # (1 unchanged attribute hidden)

      ~ config {
          + headers   = (sensitive value)
          + payload   = (sensitive value)
            # (2 unchanged attributes hidden)
        }
    }

It affects both Slack and VictorOps/Splunk-on-call notification channels. I'd also add that these fields are not present in the Terraform resource definition or in the the GraphQL API when trying to query them.

{
  "errors": [
    {
      "locations": [
        {
          "column": 15,
          "line": 11
        }
      ],
      "message": "Cannot query field \"headers\" on type \"AlertsSlackNotificationChannelConfig\"."
    },
    {
      "locations": [
        {
          "column": 15,
          "line": 12
        }
      ],
      "message": "Cannot query field \"payload\" on type \"AlertsSlackNotificationChannelConfig\"."
    }
  ]
}

Expected Behavior

No warning is given, as no changes were made inside or outside Terraform.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create a notification channel as above via Terraform
2. Run terrraform plan on a new set of changes without changing the notification channels.

Debug Output

This is from a production account so cannot post the whole content, but I have seen this in the debug output related to these channels

2022-04-04T14:38:38.567+0100 [WARN]  Provider "registry.terraform.io/newrelic/newrelic" produced an invalid plan for newrelic_alert_channel.vo-my-routingkey, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .config[0].api_key: sensitive planned value for a non-computed attribute
      - .config[0].base_url: sensitive planned value for a non-computed attribute
      - .config[0].region: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].service_key: sensitive planned value for a non-computed attribute
      - .config[0].channel: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].headers_string: sensitive planned value for a non-computed attribute
      - .config[0].payload: sensitive planned value for a non-computed attribute
      - .config[0].payload_string: sensitive planned value for a non-computed attribute
      - .config[0].recipients: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].user_id: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].teams: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].auth_password: sensitive planned value for a non-computed attribute
      - .config[0].auth_type: sensitive planned value for a non-computed attribute
      - .config[0].auth_username: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].headers: sensitive planned value for a non-computed attribute
      - .config[0].include_json_attachment: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].tags: planned value cty.StringVal("") for a non-computed attribute
      - .config[0].url: sensitive planned value for a non-computed attribute
      - .config[0].payload_type: planned value cty.StringVal("") for a non-computed attribute

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Important Factoids

We are running this over with Terragrunt.

References

Possible related to https://github.com/newrelic/terraform-provider-newrelic/issues/1324

[kidk]

Thanks @danielgblanco We will take a look. It doesn't seem like the API is returning these fields, so they are getting populated either in go-client-sdk or Terraform.

API Data

{
		"id": x,
		"name": "my-routingkey",
		"type": "victorops",
		"configuration": {
			"route_key": "my-routingkey"
		},
		"links": {
			"policy_ids": []
		}
},
{
		"id": x,
		"name": "my-channe",
		"type": "slack",
		"configuration": {
			"channel": "my-channel"
		},
		"links": {
			"policy_ids": []
		}
}

[danielgblanco]

Thanks! Let me know if you need any more info

[kidk]

This might be related to #1761 continuing research

[dvanoni]

Hey @kidk, fellow Relic here! 👋 Have you discovered anything new on this? I'm running into this same issue on version 2.44.0 of the provider.

It looks like I was able to work around the issue by adding the following to my newrelic_alert_channel resources:

lifecycle {
  ignore_changes = [
    config[0].headers,
    config[0].payload
  ]
}

Something to note is that the fix didn't come into effect until I ran an apply. After applying my config with this change, the following plan no longer included the Objects have changed outside of Terraform warning.

[dvanoni]

Hmm perhaps my workaround isn't working after all... In a follow-up to my work mentioned above, I ran an apply that created a couple more alert channel resources but then ended up failing on another resource. Since those channels did end up being creating, when I ran a subsequent plan I ran into the same warning as before. 😕