[FEATURE] Add trigger to service_principal_secret to allow rotating oauth secret

[zcking]

Use-cases

PAT tokens support auto-rotation with time_rotating as shown in the docs example: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/token#example-usage

However, this is not possible to do with databricks_service_principal_secret because it only allows passing 2 arguments: service_principal_id and lifetime, neither of which can use the .rfc3339 value of time_rotating. In other words, there is no way for users to force recreation of the oauth secret before it expires.

While the implementation does create a new secret if the state shows it as expired, this leads to scenarios where users must wait for it to expire before they can TF apply to let it create a new one, causing downtime to applications.

Attempted Solutions

Using depends_on

First I tried using an explicit depends_on to time_rotating:

resource "time_rotating" "secret_rotation" {
  rotation_days = 30
}

resource "databricks_service_principal_secret" "terraform_sp" {
  service_principal_id = databricks_service_principal.this.id
  
  lifetime = "5184000s" # Valid for 60 days

  # But recreate every 30 days
  depends_on = [
    time_rotating.secret_rotation
  ]
}

This doesn't work. It just creates the new time_rotating resource each time, but never creates the new oauth secret (until it finally expires).

Hashing expiration time and using randomness

This one is super hacky, but technically works, although it does have a very small chance of hash collision that would not trigger a rotation.

locals {
  rotation_days = 30        # How often you want the secret to actually be rotated.
  max_offset_seconds = 1440 # Increase this to lower the chance of collision. Do not go higher than rotation_days * 24 * 60 * 60

  # Hash the timestamp (changes every rotation)
  hash           = md5(time_rotating.secret_rotation.rfc3339)
  hash_as_int    = parseint(substr(local.hash, 0, 8), 16) # Convert hex to decimal
  seconds_offset = local.hash_as_int  % local.max_offset_seconds # Normalize to 0 to 1440 inclusive

  # Set the following to your desired secret lifetime, in seconds.
  # Always use a higher value than rotation_days to avoid downtime.
  # We will just use 2x the rotation_days to be safe. 
  desired_lifetime_seconds = local.rotation_days * 2 * 24 * 60 * 60
  actual_lifetime_seconds = local.desired_lifetime_seconds + local.seconds_offset
}

resource "time_rotating" "secret_rotation" {
  rotation_minutes = local.rotation_days
}

resource "databricks_service_principal_secret" "terraform_sp" {
  service_principal_id = databricks_service_principal.this.id

  # Set lifetime to the approximate desired lifetime minus the seconds_offset.
  # seconds_offset is based on the rotating time, so we can use it.
  # Note: this workaround works with some degree of randomness, with a 1 in max_offset chance of collision.
  lifetime = "${local.actual_lifetime_seconds}s"
}

Proposal

An optional triggers argument that forces recreation (same as the current expiration behavior) would solve this very cleanly, and is similar to how other terraform resources implement the behavior (e.g. null_resource).

This argument would be a list of strings, and the resource's Read operation would need to check:

1. If triggers is provided:
2. Compare triggers to the existing values stored in state file
3. If any of them have changed, force recreation

In practice this would then be used like so:

resource "time_rotating" "secret_rotation" {
  rotation_days = 30
}

resource "databricks_service_principal_secret" "terraform_sp" {
  service_principal_id = databricks_service_principal.this.id
  
  lifetime = "5184000s" # Valid for 60 days

  # But trigger rotation/recreation every 30 days
  triggers = [
    time_rotating.secret_rotation.rfc3339
  ]
}

References