terraform-provider-databricks icon indicating copy to clipboard operation
terraform-provider-databricks copied to clipboard

Published 20 hours ago • verified publisher icon databricks

[ISSUE] Unable to change owner for a `databricks_pipeline` using `databricks_permissions`

Open 840 opened this issue 1 year ago • 2 comments

There is a bug on the Databricks Platform that prevents users from changing the ownership of a DLT pipeline that publishes to Unity Catalog. Only when the DLT pipeline is run at least once, you are able to change the owner of the DLT pipeline.

Configuration

data "databricks_node_type" "node_type" {
  local_disk  = true
  min_cores   = 1
  gb_per_core = 1
}

resource "databricks_pipeline" "pipeline" {
  name    = "DLT issue"
  catalog = "dlt"

  cluster {
    label        = "default"
    node_type_id = data.databricks_node_type.node_type.id

    aws_attributes {
      first_on_demand = 1
    }

    autoscale {
      min_workers = 1
      max_workers = 2
    }
  }

  development = true

  library {
    notebook {
      path = "/Users/[email protected]/Sandbox/Hello, world"
    }
  }
}

resource "databricks_service_principal" "sp" {
  display_name = "dlt-issue"
}

resource "databricks_permissions" "usage" {
  pipeline_id = databricks_pipeline.pipeline.id

  access_control {
    service_principal_name = databricks_service_principal.sp.application_id
    permission_level       = "IS_OWNER"
  }
}

Expected Behavior

The ownership is changed to the service principal.

Actual Behavior

Terraform will state that the pipeline does not exist, even though the previous GET requests to this pipeline succeeded.

│ Error: cannot create permissions: Pipeline '986f94f5-af48-4355-bbed-8b07aa47f0e8' does not exist.
│ 
│   with databricks_permissions.usage,
│   on main.tf line 52, in resource "databricks_permissions" "usage":
│   52: resource "databricks_permissions" "usage" {

Steps to Reproduce

  1. terraform apply

Terraform and provider versions

Terraform v1.4.6 on darwin_arm64

  • provider registry.terraform.io/databricks/databricks v1.33.0

Is it a regression?

It is a regression on the Databricks Platform, not on the Databricks Terraform provider. It has already been reported internally.

Debug Output

2024-01-08T11:17:53.935+0100 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 args=[.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0]
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7807
2024-01-08T11:17:53.937+0100 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Databricks Terraform Provider
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Version 1.33.0
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: https://registry.terraform.io/providers/databricks/databricks/latest/docs
2024-01-08T11:17:54.352+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.353+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: configuring server automatic mTLS: timestamp=2024-01-08T11:17:54.352+0100
address=/var/folders/kw/4d0z66vj3l9cgff80yq691w40000gp/T/plugin432122892 network=unix timestamp=2024-01-08T11:17:54.364+0100
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7807
2024-01-08T11:17:54.383+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.383+0100 [DEBUG] ProviderTransformer: "data.databricks_node_type.node_type" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.383+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.383+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.383+0100 [DEBUG] ReferenceTransformer: "data.databricks_node_type.node_type" references: []
2024-01-08T11:17:54.383+0100 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/databricks/databricks\"]" references: []
2024-01-08T11:17:54.383+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline" references: [data.databricks_node_type.node_type]
2024-01-08T11:17:54.383+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp" references: []
2024-01-08T11:17:54.383+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage" references: [databricks_pipeline.pipeline databricks_service_principal.sp]
2024-01-08T11:17:54.386+0100 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 args=[.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0]
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7809
2024-01-08T11:17:54.388+0100 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Databricks Terraform Provider
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Version 1.33.0
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: https://registry.terraform.io/providers/databricks/databricks/latest/docs
2024-01-08T11:17:54.400+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.400+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: configuring server automatic mTLS: timestamp=2024-01-08T11:17:54.400+0100
address=/var/folders/kw/4d0z66vj3l9cgff80yq691w40000gp/T/plugin773455315 network=unix timestamp=2024-01-08T11:17:54.410+0100
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7809
2024-01-08T11:17:54.428+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.428+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.428+0100 [DEBUG] ProviderTransformer: "data.databricks_node_type.node_type (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.428+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:54.428+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline (expand)" references: [data.databricks_node_type.node_type (expand)]
2024-01-08T11:17:54.428+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp (expand)" references: []
2024-01-08T11:17:54.428+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage (expand)" references: [databricks_pipeline.pipeline (expand) databricks_service_principal.sp (expand)]
2024-01-08T11:17:54.428+0100 [DEBUG] ReferenceTransformer: "data.databricks_node_type.node_type (expand)" references: []
2024-01-08T11:17:54.428+0100 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/databricks/databricks\"]" references: []
2024-01-08T11:17:54.431+0100 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 args=[.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0]
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7810
2024-01-08T11:17:54.432+0100 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Databricks Terraform Provider
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Version 1.33.0
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: https://registry.terraform.io/providers/databricks/databricks/latest/docs
2024-01-08T11:17:54.443+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:54.443+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: configuring server automatic mTLS: timestamp=2024-01-08T11:17:54.443+0100
address=/var/folders/kw/4d0z66vj3l9cgff80yq691w40000gp/T/plugin1710519009 network=unix timestamp=2024-01-08T11:17:54.451+0100
2024-01-08T11:17:54.463+0100 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/databricks/databricks\"]" changed the config value, but that value is unused
2024-01-08T11:17:54.464+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: Explicit and implicit attributes: host, token: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/provider/provider.go:236 @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=488f22d6-e3aa-63b9-c8f2-e2f2751decf2 tf_rpc=Configure timestamp=2024-01-08T11:17:54.464+0100
2024-01-08T11:17:54.464+0100 [DEBUG] Resource instance state not found for node "databricks_service_principal.sp", instance databricks_service_principal.sp
2024-01-08T11:17:54.464+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp" references: []
2024-01-08T11:17:54.464+0100 [DEBUG] Resource instance state not found for node "data.databricks_node_type.node_type", instance data.databricks_node_type.node_type
2024-01-08T11:17:54.464+0100 [DEBUG] ReferenceTransformer: "data.databricks_node_type.node_type" references: []
2024-01-08T11:17:54.464+0100 [DEBUG] refresh: databricks_service_principal.sp: no state, so not refreshing
data.databricks_node_type.node_type: Reading...
2024-01-08T11:17:54.465+0100 [WARN]  Provider "registry.terraform.io/databricks/databricks" produced an invalid plan for databricks_service_principal.sp, but we are tolerating it because it is using the legacy plugin SDK.
      - .databricks_sql_access: planned value cty.False for a non-computed attribute
2024-01-08T11:17:55.398+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: GET /api/2.0/clusters/list-node-types
< }: tf_rpc=ReadDataSource tf_provider_addr=registry.terraform.io/databricks/databricks tf_data_source_type=databricks_node_type tf_req_id=a6180e72-b4bc-0eb7-dc53-fc1a0706f1fa @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks timestamp=2024-01-08T11:17:55.397+0100
2024-01-08T11:17:55.423+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: smallest node: m5d.large: timestamp=2024-01-08T11:17:55.423+0100
data.databricks_node_type.node_type: Read complete after 1s [id=m5d.large]
2024-01-08T11:17:55.424+0100 [DEBUG] Resource instance state not found for node "databricks_pipeline.pipeline", instance databricks_pipeline.pipeline
2024-01-08T11:17:55.424+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline" references: []
2024-01-08T11:17:55.424+0100 [DEBUG] refresh: databricks_pipeline.pipeline: no state, so not refreshing
2024-01-08T11:17:55.430+0100 [WARN]  Provider "registry.terraform.io/databricks/databricks" produced an invalid plan for databricks_pipeline.pipeline, but we are tolerating it because it is using the legacy plugin SDK.
2024-01-08T11:17:55.430+0100 [DEBUG] Resource instance state not found for node "databricks_permissions.usage", instance databricks_permissions.usage
2024-01-08T11:17:55.430+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage" references: []
2024-01-08T11:17:55.431+0100 [DEBUG] refresh: databricks_permissions.usage: no state, so not refreshing
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7810
2024-01-08T11:17:55.436+0100 [DEBUG] Resource state not found for node "databricks_service_principal.sp", instance databricks_service_principal.sp
2024-01-08T11:17:55.436+0100 [DEBUG] Resource state not found for node "databricks_pipeline.pipeline", instance databricks_pipeline.pipeline
2024-01-08T11:17:55.436+0100 [DEBUG] Resource state not found for node "databricks_permissions.usage", instance databricks_permissions.usage
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "data.databricks_node_type.node_type (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.436+0100 [DEBUG] ReferenceTransformer: "data.databricks_node_type.node_type (expand)" references: []
2024-01-08T11:17:55.436+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp" references: []
2024-01-08T11:17:55.436+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline" references: [data.databricks_node_type.node_type (expand)]
2024-01-08T11:17:55.437+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage" references: [databricks_pipeline.pipeline databricks_pipeline.pipeline databricks_pipeline.pipeline (expand) databricks_service_principal.sp databricks_service_principal.sp databricks_service_principal.sp (expand)]
2024-01-08T11:17:55.437+0100 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/databricks/databricks\"]" references: []
2024-01-08T11:17:55.437+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline (expand)" references: []
2024-01-08T11:17:55.437+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp (expand)" references: []
2024-01-08T11:17:55.437+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage (expand)" references: []
  # databricks_permissions.usage will be created
  + resource "databricks_permissions" "usage" {
  # databricks_pipeline.pipeline will be created
  + resource "databricks_pipeline" "pipeline" {
              + path = "/Users/[email protected]/Sandbox/Hello, world"
  # databricks_service_principal.sp will be created
  + resource "databricks_service_principal" "sp" {
      + databricks_sql_access      = false
2024-01-08T11:17:55.439+0100 [DEBUG] Resource state not found for node "databricks_pipeline.pipeline", instance databricks_pipeline.pipeline
2024-01-08T11:17:55.439+0100 [DEBUG] Resource state not found for node "databricks_permissions.usage", instance databricks_permissions.usage
2024-01-08T11:17:55.439+0100 [DEBUG] Resource state not found for node "databricks_service_principal.sp", instance databricks_service_principal.sp
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_permissions.usage (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "data.databricks_node_type.node_type (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_service_principal.sp" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.439+0100 [DEBUG] ProviderTransformer: "databricks_pipeline.pipeline" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databricks/databricks"]
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp (expand)" references: []
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage (expand)" references: []
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "data.databricks_node_type.node_type (expand)" references: []
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_service_principal.sp" references: []
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline" references: [data.databricks_node_type.node_type (expand)]
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_permissions.usage" references: [databricks_pipeline.pipeline databricks_pipeline.pipeline databricks_pipeline.pipeline (expand) databricks_service_principal.sp (expand) databricks_service_principal.sp databricks_service_principal.sp]
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/databricks/databricks\"]" references: []
2024-01-08T11:17:55.440+0100 [DEBUG] ReferenceTransformer: "databricks_pipeline.pipeline (expand)" references: []
2024-01-08T11:17:55.444+0100 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 args=[.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0]
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7813
2024-01-08T11:17:55.446+0100 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Databricks Terraform Provider
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Version 1.33.0
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: https://registry.terraform.io/providers/databricks/databricks/latest/docs
2024-01-08T11:17:55.466+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: 
2024-01-08T11:17:55.468+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: configuring server automatic mTLS: timestamp=2024-01-08T11:17:55.467+0100
address=/var/folders/kw/4d0z66vj3l9cgff80yq691w40000gp/T/plugin1959530949 network=unix timestamp=2024-01-08T11:17:55.478+0100
2024-01-08T11:17:55.493+0100 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/databricks/databricks\"]" changed the config value, but that value is unused
2024-01-08T11:17:55.493+0100 [INFO]  provider.terraform-provider-databricks_v1.33.0: Explicit and implicit attributes: host, token: tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=56bb42fb-915e-32bb-24d5-3da5ab601add tf_rpc=Configure @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/provider/provider.go:236 @module=databricks timestamp=2024-01-08T11:17:55.493+0100
2024-01-08T11:17:55.494+0100 [WARN]  Provider "registry.terraform.io/databricks/databricks" produced an invalid plan for databricks_service_principal.sp, but we are tolerating it because it is using the legacy plugin SDK.
      - .databricks_sql_access: planned value cty.False for a non-computed attribute
databricks_service_principal.sp: Creating...
2024-01-08T11:17:55.494+0100 [INFO]  Starting apply for databricks_service_principal.sp
2024-01-08T11:17:55.494+0100 [DEBUG] databricks_service_principal.sp: applying the planned Create change
2024-01-08T11:17:55.497+0100 [WARN]  Provider "registry.terraform.io/databricks/databricks" produced an invalid plan for databricks_pipeline.pipeline, but we are tolerating it because it is using the legacy plugin SDK.
databricks_pipeline.pipeline: Creating...
2024-01-08T11:17:55.497+0100 [INFO]  Starting apply for databricks_pipeline.pipeline
2024-01-08T11:17:55.497+0100 [DEBUG] databricks_pipeline.pipeline: applying the planned Create change
2024-01-08T11:17:56.446+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: POST /api/2.0/pipelines
>         "path": "/Users/[email protected]/Sandbox/Hello, world"
< }: tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=b4f48e8c-97f8-de79-57e3-80f21474b340 tf_resource_type=databricks_pipeline tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks timestamp=2024-01-08T11:17:56.446+0100
2024-01-08T11:17:56.446+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: Waiting for state to become: [success]: timestamp=2024-01-08T11:17:56.446+0100
2024-01-08T11:17:56.678+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: GET /api/2.0/pipelines/986f94f5-af48-4355-bbed-8b07aa47f0e8
<   "creator_user_name": "[email protected]",
<   "run_as_user_name": "[email protected]",
<           "path": "/Users/[email protected]/Sandbox/Hello, world"
< }: tf_provider_addr=registry.terraform.io/databricks/databricks tf_resource_type=databricks_pipeline @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_req_id=b4f48e8c-97f8-de79-57e3-80f21474b340 tf_rpc=ApplyResourceChange @module=databricks timestamp=2024-01-08T11:17:56.678+0100
2024-01-08T11:17:56.911+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: GET /api/2.0/pipelines/986f94f5-af48-4355-bbed-8b07aa47f0e8
<   "creator_user_name": "[email protected]",
<   "run_as_user_name": "[email protected]",
<           "path": "/Users/[email protected]/Sandbox/Hello, world"
< }: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_resource_type=databricks_pipeline @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=b4f48e8c-97f8-de79-57e3-80f21474b340 tf_rpc=ApplyResourceChange timestamp=2024-01-08T11:17:56.911+0100
2024-01-08T11:17:56.912+0100 [WARN]  Provider "provider[\"registry.terraform.io/databricks/databricks\"]" produced an unexpected new value for databricks_pipeline.pipeline, but we are tolerating it because it is using the legacy plugin SDK.
databricks_pipeline.pipeline: Creation complete after 2s [id=986f94f5-af48-4355-bbed-8b07aa47f0e8]
2024-01-08T11:17:57.005+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: POST /api/2.0/preview/scim/v2/ServicePrincipals
< }: tf_resource_type=databricks_service_principal tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=23f63901-0938-4a14-c01a-a11cd6106c94 tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks timestamp=2024-01-08T11:17:57.005+0100
2024-01-08T11:17:57.245+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: GET /api/2.0/preview/scim/v2/ServicePrincipals/7612343260391692?attributes=userName,displayName,active,externalId,entitlements
< }: @module=databricks tf_req_id=23f63901-0938-4a14-c01a-a11cd6106c94 tf_resource_type=databricks_service_principal tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/databricks/databricks @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 timestamp=2024-01-08T11:17:57.245+0100
2024-01-08T11:17:57.245+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: read SP '7612343260391692': {7612343260391692 [] dlt-issue true [urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal]  71e25b42-5e0b-4c20-81e7-e486a9aa7bd3 [] map[] [] [] }: timestamp=2024-01-08T11:17:57.245+0100
databricks_service_principal.sp: Creation complete after 2s [id=7612343260391692]
databricks_permissions.usage: Creating...
2024-01-08T11:17:57.259+0100 [INFO]  Starting apply for databricks_permissions.usage
2024-01-08T11:17:57.260+0100 [DEBUG] databricks_permissions.usage: applying the planned Create change
2024-01-08T11:17:58.202+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: GET /api/2.0/preview/scim/v2/Me
<       "value": "[email protected]"
<       "value": "databricks-sql-access"
<   "userName": "[email protected]"
< }: tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=623c7c0b-863f-a6e4-6148-623efec0ff30 tf_resource_type=databricks_permissions @module=databricks timestamp=2024-01-08T11:17:58.202+0100
2024-01-08T11:17:58.925+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: non-retriable error: Pipeline '986f94f5-af48-4355-bbed-8b07aa47f0e8' does not exist.: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_rpc=ApplyResourceChange tf_resource_type=databricks_permissions @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=623c7c0b-863f-a6e4-6148-623efec0ff30 timestamp=2024-01-08T11:17:58.925+0100
2024-01-08T11:17:58.925+0100 [DEBUG] provider.terraform-provider-databricks_v1.33.0: PUT /api/2.0/permissions/pipelines/986f94f5-af48-4355-bbed-8b07aa47f0e8
< }: tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=623c7c0b-863f-a6e4-6148-623efec0ff30 tf_resource_type=databricks_permissions @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_rpc=ApplyResourceChange @module=databricks timestamp=2024-01-08T11:17:58.925+0100
2024-01-08T11:17:58.925+0100 [ERROR] provider.terraform-provider-databricks_v1.33.0: Response contains error diagnostic: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:62 @module=sdk.proto diagnostic_detail= diagnostic_summary="cannot create permissions: Pipeline '986f94f5-af48-4355-bbed-8b07aa47f0e8' does not exist." tf_proto_version=5.4 tf_req_id=623c7c0b-863f-a6e4-6148-623efec0ff30 tf_resource_type=databricks_permissions diagnostic_severity=ERROR tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange timestamp=2024-01-08T11:17:58.925+0100
2024-01-08T11:17:58.931+0100 [ERROR] vertex "databricks_permissions.usage" error: cannot create permissions: Pipeline '986f94f5-af48-4355-bbed-8b07aa47f0e8' does not exist.
  with databricks_permissions.usage,
  on main.tf line 52, in resource "databricks_permissions" "usage":
  52: resource "databricks_permissions" "usage" {
path=.terraform/providers/registry.terraform.io/databricks/databricks/1.33.0/darwin_arm64/terraform-provider-databricks_v1.33.0 pid=7813

Important Factoids

-

Would you like to implement a fix?

-

840 avatar Jan 08 '24 10:01 840

cc @zshao9 this is probably what is affecting your pipeline permissions resources.

mgyucht avatar Jan 08 '24 14:01 mgyucht

Hi @840 Thanks for raising this. We are currently working on a fix. Until then, and as you say, you will need to run the Pipeline once before you can change the permissions.

hectorcast-db avatar Jan 18 '24 15:01 hectorcast-db