[ISSUE] mlflow_webhooks resource fails with permission errors during resource creation, when listing all webhooks

[smurching]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Databricks Provider issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Configuration

resource "databricks_mlflow_webhook" "model_deployment_webhook" {
  model_name = databricks_mlflow_model.registered_model.name
  events      = ["MODEL_VERSION_CREATED"]
  description = "Trigger model deployment workflow when model training and registration completes"
  http_url_spec {
    url = "https://api.github.com/repos/${var.github_repo}/actions/workflows/deploy-model.yml/dispatches"
    authorization = "token ${var.github_actions_token}"
  }
}

Expected Behavior

I expected the model registry webhook specified above to be created

Actual Behavior

When attempting to get a particular registry webhook, the Databricks Terraform Provider registry webhooks resource attempts to list all registry webhooks and find the webhook with the specified ID - note that we don’t yet have a /registry-webhooks/get endpoint to get registry webhooks by ID.

This typically fails when run as a service principal because listing all registry webhooks requires CAN_MANAGE permission on the model registry (?), to list registry-wide webhooks:

Error: cannot read mlflow webhook: error reading list of webhooks: Request ListRegistryWebhooks failed access control checks. User has permission CAN_READ but requires one of CAN_MANAGE. Using pat auth: host=https://adb-8058959194909749.9.dev.azuredatabricks.net/, token=***REDACTED***

The quickest fix is likely to supply the model name when fetching the registry webhook, if the field is available/non-null.

Steps to Reproduce

Invoke terraform apply on the above config

[smurching]

@nfx FYI - is this something you may be able to get to? No worries if not, thanks! cc @arpitjasa-db

[nfx]

@smurching Is there an API to get a single webhook without listing all of them?

[alexott]

There is no such API to get information about specific webhook

[smurching]

@nfx unfortunately not, as Alex mentioned. We definitely should add one (in general my learning from this is that our REST APIs need to support the full typical range of CRUD operations to be Terraform-friendly), but hoping to fix this in the meantime by supplying model name to the /registry-webhooks/list endpoint when available

[nfx]

@smurching i'm closing this issue then, please create a feature follow-up for webhooks, as it's not possible to fix it without CRUD. please reopen it once API is available

[smurching]

@nfx we're happy to ping again once the REST API is available. Any objection to us unblocking ourselves in the near term though by sending a PR (we can write it ourselves) for this:

hoping to fix this in the meantime by supplying model name to the /registry-webhooks/list endpoint when available

Adding a new GET REST API is nontrivial work & will take some time to rollout, unfortunately, and currently we/customers are blocked on using this resource

[smurching]

Discussed offline with @nfx - we'll send the fix PR ourselves in the meantime given that we don't yet have a GET API for registry webhooks

[nfx]

Following up - is this issue still relevant?

[arpitjasa-db]

@nfx yeah the resource still fails without admin permissions

[nfx]

Thank you for the feature request! Currently, the team operates in a limited capacity, carefully prioritizing, and we cannot provide a timeline to implement this feature. Please make a Pull Request if you'd like to see this feature sooner, and we'll guide you through the journey.