Error when updating token resource in 0.3.10

[kpucynski]

Describe the bug

In the new provider version 0.3.10 updating dbtcloud_service_token resource is failing.

Error message

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ dbtcloud_service_token.tokens["token_name"],
│ provider "provider[\"registry.terraform.io/dbt-labs/dbtcloud\"]" produced
│ an unexpected new value: .service_token_permissions: planned set element
│ cty.ObjectVal(map[string]cty.Value{"all_projects":cty.False,
│ "permission_set":cty.StringVal("developer"),
│ "project_id":cty.NumberIntVal(3597),
│ "writable_environment_categories":cty.SetVal([]cty.Value{cty.StringVal("all")})})
│ does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Resource configuration

Service tokens are configured with the loop. Affected token have around 50 project IDs assigned.

resource "dbtcloud_service_token" "tokens" {
  for_each = local.tokens

  name = each.key

  dynamic "service_token_permissions" {
    for_each = each.value
    content {
      permission_set = "developer"
      all_projects   = false
      project_id     = service_token_permissions.value
    }
  }

  dynamic "service_token_permissions" {
    for_each = each.value
    content {
      permission_set = "git_admin"
      all_projects   = false
      project_id     = service_token_permissions.value
    }
  }

  dynamic "service_token_permissions" {
    for_each = each.value
    content {
      permission_set = "job_admin"
      all_projects   = false
      project_id     = service_token_permissions.value
    }
  }
}

Expected behavior

Token updated in place.

Config (please complete the following information):

• Installed dbt-labs/dbtcloud v0.3.10 (self-signed, key ID 644DCEA333782266)

Additional context

Downgrade to version 0.3.8 solves the problem and update is passing.

[b-per]

Thanks for the issue. We will try to reproduce it.

Could you please tell us what dbt Cloud tenant you are on? US MT, EMEA MT, ST (Single Tenant)?

[kpucynski]

We are using EMEA: https://emea.dbt.com/api

[chasewalden]

Thanks @kpucynski . Looking into this now. This appears to be a bug with a new feature that is conditionally enabled.

[b-per]

Do you still have an issue with the latest releases? It might have gone now as the env level permissions should be active for all accounts.

[b-per]

Closing for now. We can reopen if there is still an issue.

[kpucynski]

@b-per, We have just tested code with new 0.3.22 provider version and get the same issue:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ dbtcloud_service_token.dcrt_tokens["token"],
│ provider "provider[\"registry.terraform.io/dbt-labs/dbtcloud\"]" produced
│ an unexpected new value: .service_token_permissions: planned set element
│ cty.ObjectVal(map[string]cty.Value{"all_projects":cty.False,
│ "permission_set":cty.StringVal("developer"),
│ "project_id":cty.NumberIntVal(3190),
│ "writable_environment_categories":cty.SetVal([]cty.Value{cty.StringVal("all")})})
│ does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[kpucynski]

0.3.8 still no issues.

[kpucynski]

@b-per can you reopen the issue?

[b-per]

I just reopened it but we haven't been able to reproduce it. We can try again.

[b-per]

I just retried your code with 10 projects on the latest provider

locals {
  tokens = {
    "issue280": [
        1234,
        2345,
        ... (10 in total)
    ]
  }
}

And I can't reproduce the issue. The token gets created and I can remove/add projects without a problem.

Could you please share your dbt Cloud Account ID?

[b-per]

Good/Bad news. I created 40 projects on the fly and can now reproduce the issue when trying to assign permissions for 40 projects!

[b-per]

@chasewalden

The issue here is that when a lot of permissions are assigned to a token, we don't return all of them and actually need to loop through the paginated results. So, we need to add pagination in GetServiceTokenPermissions.

https://github.com/dbt-labs/terraform-provider-dbtcloud/blob/cc7fa2db09f374cccf5c2d2400de6c4e50c23373/pkg/dbt_cloud/service_token.go#L47-L65

this is the reply we get when we query a token with more than 100 permissions

...
            "account_id": 1234,
            "all_projects": false,
            "created_at": "2024-12-11 13:45:57.925949+00:00",
            "id": 532831,
            "permission_set": "job_admin",
            "project_id": 1111,
            "service_token_id": 22222,
            "state": 1,
            "updated_at": "2024-12-11 13:45:57.925963+00:00",
            "writable_environment_categories": []
        }
    ],
    "error_code": null,
    "extra": {
        "filters": {
            "account_id": 1234,
            "limit": 100,
            "offset": 0,
            "service_token_id": 22222,
            "state": "active"
        },
        "order_by": null,
        "pagination": {
            "count": 100,
            "total_count": 120
        }
    },
    "status": {
        "code": 200,
        "developer_message": "",
        "is_success": true,
        "user_message": "Success!"
    }
}

I checked the permissions for groups and it looks that endpoints are slightly different so we might not have the same problem there.

Can you or the team work on a fix here or do you want me to do it?

[kpucynski]

@b-per we have currently 52 projects on the problematic list, so one resource will create over 150 permissions. I would like to not post any project related details here - ping me where(if) I can contact you with the details.

[b-per]

Hi @kpucynski . All good. We have all the details to reproduce it so we don't need more inputs for now. We just need to implement the fix.

[ashtonwalden]

@kpucynski, we have added a fix for this in v0.3.23