Very slow resource refreshing when not found

[dbaggott]

I'm using this provider to manage rollbar notifications and it's taking an extremely long time to refresh notifications as can be seen from the timestamps in this output from the planning phase:

module.rollbar.module.main_qa_notifications.module.content-service_notifications.rollbar_notification.slack["new_item"]: Refreshing state... [id=7300069]
2025-03-21T16:57:46.428-0700 [DEBUG] provider.terraform-provider-rollbar_v1.15.1: {"level":"debug","notificationID":7300069,"time":"2025-03-21T16:57:46-07:00","message":"Reading notification from API"}
2025-03-21T16:59:02.939-0700 [DEBUG] provider.terraform-provider-rollbar_v1.15.1: {"level":"error","notificationID":7300069,"error":"not found","time":"2025-03-21T16:59:02-07:00"}

Note that it appears to have taken over a minute and that there is a "not found" message. We have lots of notifications and so this is taking an hour plus to refresh them all!

From looking at the code, it looks like there is retry logic implemented and that it kicks in if a 404 not found is returned.

I believe the slowness is b/c we had a bunch of notifications removed via the UI and so the provider is 404ing when trying to fetch resources via the api which is causing retries to happen up to the retry limit.

[linear[bot]]

SDK-459 Very slow resource refreshing when not found

[dbaggott]

(My suggestion is to remove retry logic on 404s as that's a valid response and can be expected.)

[brianr]

Thanks @dbaggott , we'll take a look.

[brianr]

@dbaggott , we're discussing this. Wondering - what's the desired behavior here - as we understand it, this arises when a rule has been removed outside of Terraform (i.e. through the UI). Is your goal to update your terraform config to reflect the changes made in the UI? Or to re-create this rule in the UI?

[dbaggott]

Hi, the expected behavior is that terraform will update Rollbar so that the rules that were manually deleted are recreated. In other words, after the terraform apply completes, the remote rollbar state will match the terraform configuration. That part is working as expected.

The only issue here is during the planning phase. When the rollbar provider checks to see if the expected rule exists, the rollbar server correctly returns a 404 indicating that the rule does not exist. This is expected and makes sense. The problem is that the rollbar provider treats a 404 the same as an intermittent error (such as a 5xx) and goes into a retry loop. It keeps on retrying, keeps on getting the correct 404 response, and repeats this up until the rollbar provider's retry maximum. When you have lots of rules that all got deleted, this can take hours to go through them all!

The suggested change is to not invoke the retry logic on a 404 response.

On Wed, Apr 9, 2025 at 2:44 PM Brian Rue @.***> wrote:

@dbaggott https://github.com/dbaggott , we're discussing this. Wondering - what's the desired behavior here - as we understand it, this arises when a rule has been removed outside of Terraform (i.e. through the UI). Is your goal to update your terraform config to reflect the changes made in the UI? Or to re-create this rule in the UI?

— Reply to this email directly, view it on GitHub https://github.com/rollbar/terraform-provider-rollbar/issues/421#issuecomment-2791060336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA3QJIFT5X5DKPU4NVI7WL2YWIFTAVCNFSM6AAAAABZRED6E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJRGA3DAMZTGY . You are receiving this because you were mentioned.Message ID: @.***> brianr left a comment (rollbar/terraform-provider-rollbar#421) https://github.com/rollbar/terraform-provider-rollbar/issues/421#issuecomment-2791060336

@dbaggott https://github.com/dbaggott , we're discussing this. Wondering - what's the desired behavior here - as we understand it, this arises when a rule has been removed outside of Terraform (i.e. through the UI). Is your goal to update your terraform config to reflect the changes made in the UI? Or to re-create this rule in the UI?

— Reply to this email directly, view it on GitHub https://github.com/rollbar/terraform-provider-rollbar/issues/421#issuecomment-2791060336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA3QJIFT5X5DKPU4NVI7WL2YWIFTAVCNFSM6AAAAABZRED6E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJRGA3DAMZTGY . You are receiving this because you were mentioned.Message ID: @.***>