400 error about `locked` attribute but we aren't using it

[martinb3]

Datadog Terraform Provider Version

v3.37.0

Terraform Version

v1.5.7

What resources or data sources are affected?

datadog_monitor

Terraform Configuration Files

resource "datadog_monitor" "error_rate_monitor" { 
  name    = "[${var.env}] ${var.name} - ${var.service_metric_path}/${var.method} error rate sub ${var.max_error_rate_pct}% - backing monitor"
  type    = "metric alert"
  message = "The error rate for `${var.service_metric_path}/${var.method}` has exceeded ${var.max_error_rate_pct}%"

  query = "sum(${var.time_window}):${local.numerator} / ${local.denominator} * 100 > ${var.max_error_rate_pct}"

  force_delete        = false
  include_tags        = true
  notify_no_data      = var.notify_no_data
  no_data_timeframe   = var.no_data_timeframe
  require_full_window = false
  validate            = true
  timeout_h           = 1

  restricted_roles = var.restricted_roles

  tags = concat(var.common_tags, var.additional_tags, local.metadata_tags)
}

Relevant debug or panic output

400 Bad Request: {"errors":["The `locked` monitor option has been replaced with role-based access control: https://docs.datadoghq.com/monitors/guide/how-to-set-up-rbac-for-monitors/?tab=ui#migrate-monitors-from-locked-to-restricted-roles"]}

Expected Behavior

Because we aren't using locked, we don't expect an error.

Actual Behavior

We receive a rejection from the Datadog API.

400 Bad Request: {"errors":["The `locked` monitor option has been replaced with role-based access control: https://docs.datadoghq.com/monitors/guide/how-to-set-up-rbac-for-monitors/?tab=ui#migrate-monitors-from-locked-to-restricted-roles"]}

Steps to Reproduce

No locked attribute has been specified, however it is present in the state file (likely from previous run?).

Important Factoids

Use an older provider at first, so that locked is in your state.

References

https://github.com/DataDog/terraform-provider-datadog/pull/1400 https://docs.datadoghq.com/monitors/guide/how-to-set-up-rbac-for-monitors/?tab=ui#migrate-monitors-from-locked-to-restricted-roles

[phillip-dd]

Hi @martinb3, thanks for opening an issue. This is not likely related to #1400 directly.

All Datadog organizations making API requests where the locked option was true were notified of a 24 hour "brownout" affecting those API requests.

We're assessing the case described here and determining next steps. The intent is not to block the creation/editing of monitors where locked was previously set to true, but no longer is. Can you verify that all of the monitors you're editing fall into that case?

[oeuftete]

@phillip-dd As someone else running into this issue, I can provide our team's experience.

Terraform plans are blocked for some monitors where locked was set to true in the terraform state, though the value was not set explicitly in the terraform configuration; we moved to roles a very long time ago. I'm not sure how we got into this state... possibly through an earlier import, or previous provider behaviour not disabling the attribute when we shifted to using restricted roles instead.

We've worked around the issue by tainting and rebuilding the affected monitors; their terraform states still have the locked attribute, but it is now false, and the errors no longer occur.

[martinb3]

@phillip-dd hi there, thanks for that additional information! I'm not 100% sure we got notified, but I can confirm we aren't setting locked right now on the monitor that's affected's terraform config (not sure historically, would have to really search through a LOT of commit data to tell you for sure). In our state, locked is false, for the affected monitors -- which seems odd if tainting (as @oeuftete said) fixes it. But we did delete one monitor and recreate it, and confirm it did resolve the issue. We just can't manually do that with thousands of monitors (and not to mention losing monitor history). Thank you!

edit: I did find that we previously set locked = true for some of these monitors, where it isn't set now. But in the state file, there's still some instances of locked: true in the JSON.

[SimoneT-DD]

Hi @martinb3, thanks for providing extra details.

But in the state file, there's still some instances of locked: true in the JSON.

This is definitely what is causing the issue and we are actively looking into this problem.

Thanks, Simone

[joedenniss]

An alternative workaround that seems to work is to edit the monitor through the Datadog UI and save it without making any changes, then reapply.

[phillip-dd]

Closing this issue - locked: true (without restricted_roles) is no longer a valid attribute to pass for monitors. To support backward compatibility locked: true (with restricted_roles) and locked: false` will continue to be accepted.