[FEATURE] Add strongly typed data source for writing databricks_cluster_policy definitions

[zcking]

Use-cases

Writing databricks cluster policies is very semi-structured and loosely typed. This makes it difficult to write effective cluster policies on the first try, and users end up writing, praying, testing, tweaking, more praying, so on and so forth. Even in Terraform, the user experience is bad: here's a definition argument, just dump a JSON blob into it and hope you get it right.

We should strive to provide an interface that maps to the policy data structure, which would make it easier to write correct policies without all the guesswork (even with documentation).

Attempted Solutions

You could define your own typed variable that aims to model the policy definition:

variable "policy_attributes" {
  type = list(object({
    type = string
    isOptional = optional(bool)
    value = optional(string)
    defaultValue = optional(string)
    pattern = optional(string)
    isHidden = optional(bool)
...
  }))
}

But this is fragile, very error-prone, and can't offer good validation of those values.

I have even taken to writing my own app https://github.com/zcking/dbx-policy-builder but I would love to see this feature be addressed in the terraform provider.

Proposal

Ideally we can introduce a new data source databricks_cluster_policy_definition which takes the same approach as aws_iam_policy_document. This would let users write a policy definition with strongly typed HCL in the data source, then JSON-encode it when passing into the existing databricks_cluster_policy resource.

This is what it would look like in practice:

data "databricks_cluster_policy_definition" "ml_interactive_small" {  
  attribute {
    name = "cluster_type"
    type = "fixed"
    value = "all-purpose"
  }

  attribute {
    name = "workload_type.clients.notebooks"
    type = "fixed"
    value = true
  }

  attribute {
    name = "workload_type.clients.jobs"
    type = "fixed"
    value = false
  }

  attribute {
    name = "dbus_per_hour"
    type = "range"
    minValue = 0
    maxValue = 4
  }

  attribute {
    name = "runtime_engine"
    type = "fixed"
    value = "STANDARD"
    hidden = true
  }

  # ...
}

resource "databricks_cluster_policy" "ml_interactive_small" {
  name = "Interactive ML (Small)"
  description = "Used for creating small single user ML clusters for interactive notebooks only."
  definition = data.databricks_cluster_policy_definition.ml_interactive_small.json
}

References