Pagerduty terraform failed to create event rule if the existing is deleted

[tgyogesh]

Hi there,

Pagerduty terraform failed to create event rule if the existing one is deleted manually. Execute terraform with some event rules in existing ruleset, delete one event rule and re-execute the terraform with the previous state and verify the event rule is created or not.

Terraform Version

Terraform v0.15.3 on windows_amd64

Affected Resource(s)

Please list the resources as a list, for example: eventrules

Terraform Configuration Files

resource "pagerduty_ruleset_rule" "Application_Open_Rule" { ruleset = data.pagerduty_ruleset.global_ruleset.id conditions { operator = "and" subconditions { operator = "contains" parameter { value = var.service_name path = "payload.custom_details.Tags" } } subconditions { operator = "nequals" parameter { path = "payload.custom_details.Impact" value = "INFRASTRUCTURE" } } subconditions { operator = "equals" parameter { path = "payload.custom_details.State" value = "OPEN" } } } actions { route { value = pagerduty_service.Application_Service.id } } }

Debug Output

│ Error: GET API call to https://api.pagerduty.com/rulesets/975523db-0915-496d-9e54-4e6e1cf2e440/rules/59ad4cbc-7fdc-44b7-b212-0bce74e58ca5 (https://api.pagerduty.com/rulesets/975523db-0915-496d-9e54-4e6e1cf2e440/rules/59ad4cbc-7fdc-44b7-b212-0bce74e58ca5) failed 404 Not Found. Code: 0, Errors: , Message: Rule Not Found

Panic Output

2021-08-11T09:20:42.998+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: 2021/08/11 09:20:42 [DEBUG] PagerDuty API Response Details: 2021-08-11T09:20:42.999+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: ---[ RESPONSE ]-------------------------------------- 2021-08-11T09:20:42.999+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: HTTP/2.0 404 Not Found 2021-08-11T09:20:42.999+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Content-Length: 38 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Access-Control-Allow-Origin: * 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Cache-Control: max-age=0, private, must-revalidate 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Content-Type: application/vnd.pagerduty+json; charset=UTF-8 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Date: Wed, 11 Aug 2021 03:50:42 GMT 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: Server: nginx 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: X-Request-Id: 8e3e2430-ab64-4567-83f1-eee83c96067f 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: { 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: "error": { 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: "message": "Rule Not Found" 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: } 2021-08-11T09:20:43.000+0530 [DEBUG] provider.terraform-provider-pagerduty_v1.9.6.exe: }

Expected Behavior

If the terraform pagerduty detects the rule is not present then it should recreate until the actual logic of terraform script is changed.

Actual Behavior

It checks for the existing event rule and throws rule not found error.

[PrzemyslawSpringer]

Can anyone look at that? The similar happens to us. Today I tried to deploy 3 services with 5 event rules each, having a clear, not deployed earlier environment. On two event rules in one service I got 404 from PagerDuty API during 'apply' and those rules are now "tainted" in the tf state. The same happened to my colleague few days ago. I cannot 'apply' again nor 'destroy' - always get 404 on those rules. Tried 'tf plan -replace="x"' - 404.

I've dealt with that once by removing the missing rules manually from the tf state, but it was a pain. And now it happened on my production environment, so it's even a bigger pain.

Terraform 1.0 PagerDuty provider 2.2.1 S3 remote state - if that's relevant

EDIT: I managed to deal with my issue kind of automatically. May be of use to someone:

terraform init
terraform state pull > state.json
jq -r "del(.resources[] | select(.type==\"pagerduty_service_event_rule\") | .instances[] | select(.status==\"tainted\"))" state.json > fixed.json
terraform state push -force fixed.json
terraform plan -out=plan
terraform apply plan

Still, it's some nasty modification of a state file. Terraform generally tries to prevent messing a state up with those state commands, but force disables the checks, so no idea what may happen in other case.

[abeluck]

This is pretty killer. A user made a mistake and deleted an event rule in the PD UI, and now our entire PD terraform automation is broken :(

[netjordan]

This is happening acrosss lots of resources with this provider - there's another issue #354 for this. Please can we thumbs up that issue, and hopefully the PagerDuty team will prioritise that work to resolve this issue.

[gsreynolds]

Rules are being deprecated and replaced

[arivictor]

@gsreynolds nice, so just "good luck everyone" until then?