[DOC] databricks_artifact_allowlist usage unclear

[sebolabs]

Affected Resource(s)

https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/artifact_allowlist

Expected Details

The docs say:

This resource could be only used with workspace-level provider!

So that would mean that you define that resource for every workspace you create in your account.

HOWEVER, that's not the case because having that resource defined per workspace where a workspace in our case represents an environment, whenever we run Terraform it wants to overwrite the allowlist which means this setting is global and should be considered account-level.

This statement:

It is required to define all allowlist for an artifact type in a single resource, otherwise Terraform cannot guarantee config drift prevention.

doesn't make it clear. It would if that resource was used with the account-level provider.

List of things to potentially add/remove

You decide to make it clear for people.

Important Factoids

References

[nkvuong]

@sebolabs thank you for raising this. I agree that the doc reads confusing, below is the explanation:

• Artifact allowlist must be managed via workspace-level API, not account-level API (first note). This is similar to other UC objects
• However, the artifact allowlist is a metastore-level object that only supports full replacement (PUT instead of PATCH), so changes will apply across the entire metastore (second note). This is similar to databricks_grants where only a single resource per object is allowed.

Happy to consider ways to make this clearer for end users

[sebolabs]

Hi @nkvuong and thanks for your response.

While docs can always be made clearer I don't understand why the allow list that relates to the following artifcat types INIT_SCRIPT | LIBRARY_JAR | LIBRARY_MAVEN is considered to be a UC-level setting. They are meant to be leveraged at the compute layer and that pretty much equals clusters running in workspaces.

Then, purely from the Terraform provider perspective, to get that allowlist set up your provider configuration must have a host configured that is your workspace endpoint URL. In our case when we have multiple environments having their dedicated workspaces we must either conditionally apply that resource as it can be defined just once or have it defined elsewhere but still pick one workspace and use its URL in the provider configuration.

I hope you can see what I'm saying... you use the Account API to create a metastore but then you use the Workspace API to configure the allow list that is a metasore-level object while in fact init scripts run on clusters belonging to workspaces 🙃

[atrbgithub]

HOWEVER, that's not the case because having that resource defined per workspace where a workspace in our case represents an environment, whenever we run Terraform it wants to overwrite the allowlist which means this setting is global and should be considered account-level.

We're seeing the exact same thing.

The docs state:

This resource could be only used with workspace-level provider!

Lets say you have preprod and prod workspaces, and you have a block of TF like this:

resource "databricks_artifact_allowlist" "init_script_allowlist" {
  provider      = databricks.workspace
  artifact_type = "INIT_SCRIPT"
  artifact_matcher {
    artifact   = "/Volumes/some-location-${var.environment}/some-other-path/"
    match_type = "PREFIX_MATCH"
  }
}

When you push this TF to preprod, the allow list is created with the path:

/Volumes/some-location-preprod/some-other-path/

When you then push to prod, the same allow list is updated, a second one is not added which is what I believe users would expect the code to do.

So then you end up with a single allow list with this as the path:

/Volumes/some-location-prod/some-other-path/

That means that one of the terraform runs will have to manage the allow list for both workspaces.

[philippbussche]

Hi @nkvuong and thanks for your response.

While docs can always be made clearer I don't understand why the allow list that relates to the following artifcat types INIT_SCRIPT | LIBRARY_JAR | LIBRARY_MAVEN is considered to be a UC-level setting. They are meant to be leveraged at the compute layer and that pretty much equals clusters running in workspaces.

Then, purely from the Terraform provider perspective, to get that allowlist set up your provider configuration must have a host configured that is your workspace endpoint URL. In our case when we have multiple environments having their dedicated workspaces we must either conditionally apply that resource as it can be defined just once or have it defined elsewhere but still pick one workspace and use its URL in the provider configuration.

I hope you can see what I'm saying... you use the Account API to create a metastore but then you use the Workspace API to configure the allow list that is a metasore-level object while in fact init scripts run on clusters belonging to workspaces 🙃

I agree with @sebolabs about the inconsistency when it comes to API usage, we have experienced it somewhere else as well, when having to configure external locations. This is also something to be done on the metastore, through a specific workspace. In our environment we also have separate workspaces and want to de-centralize management/administration where this now becomes a problem.

However it feels we have several issues at hand here:

1. API availability on workspace level where you would expect it to be available through the account
2. Applicable object type is metastore where you would expect it to be compute/cluster (which does not exist in UC I think)
3. General permission inconsistency where you can install pypi packages on a cluster when you have the CAN_MANAGE permission on compute level but where you need the UC permission on metastore for maven packages
4. Potential inconsistency where you can install a maven package without any additional permissions if you have a UC enabled cluster that is running in single user access mode but you cannot if it is UC enabled and using the shared access mode.

Coming back to our initial challenge with external locations which are also to be configured on metastore level but through an individual workspace we did find a solution (for now) which is that we abstracted (custom TF module) the usage of the databricks provider so that we allow teams to configure external locations through any workspace code they want to and then our TF module will figure out if the external location already exists but was created somewhere else or if the external location still has to be created. We are using data resource for lookups and also the comment field to determine if external locations were created already but we have seen that this is not 100% (thread) safe.

So for us we have somewhat solved the Account API vs. Workspace API issue but with the current issue on artifact allowlists we would still have the problem that we want to define the permission (ACL or UC) on cluster level or worst case on workspace level. It does not make sense to us that this (for maven artifacts) is now something that needs to be configured on UC/metastore level. Again this is due to our approach for de-centralizing admin capabilities. Admins of one workspace should be able to change these things for clusters belonging to their workspace but if this is set on metastore level this applies to any cluster eventually ?!?

[simonvanderveldt]

Just wanted to check, I assume none of this would be relevant if it would just work to use init scripts from the workspace in standard (formerly shared) access mode, is that correct? Since it's in the workspace it's automatically workspace specific solving all the issues with this being centrally managed because of the unity catalog usage, however for some reason this isn't supported in standard access mode according to the docs.