cli icon indicating copy to clipboard operation
cli copied to clipboard
verified publisher icon databricks

[Azure] Unity Catalog: Perform permission check during Storage Credential creation instead of role ID lookup

Open mixam24 opened this issue 1 year ago • 3 comments

Describe the issue

I'm trying to create a new storage credential in Unity Catalog using my personal user that has Unity Metastore Admin permissions. Due to security reasons, our organization does not allow personal users to have the built-in "Contributor" role. Instead, we assign a custom one, which is almost identical to the built-in role. Hence, my personal identity does not have built-in "Contributor" role.

The custom role definition is as follows:

{
  "assignableScopes": [
    "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
    "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
    "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
  ],
  "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
  "createdOn": "2019-07-24T12:59:19.599048+00:00",
  "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
  "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
  "permissions": [
    {
      "actions": ["*"],
      "condition": null,
      "conditionVersion": null,
      "dataActions": [],
      "notActions": [
        "Microsoft.Authorization/*/Delete",
        "Microsoft.Authorization/*/Write",
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Blueprint/blueprintAssignments/write",
        "Microsoft.Blueprint/blueprintAssignments/delete",
        "Microsoft.Compute/galleries/share/action",
        "Microsoft.Purview/consents/write",
        "Microsoft.Purview/consents/delete",
        "Microsoft.Subscription/rename/action",
        "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
      ],
      "notDataActions": []
    }
  ],
  "roleName": "Custom Contributor",
  "roleType": "CustomRole",
  "type": "Microsoft.Authorization/roleDefinitions",
  "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
  "updatedOn": "2024-05-16T08:53:05.530192+00:00"
}

The attempt to create a storage credential fails with the following error: Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

I believe that the service performs role assignment lookups only by ID and does not check that my user has the necessary permissions to perform the action.

Steps to reproduce the behavior

  1. List the identity role assignments over the databricks access connector 
    ~/ ❯ az role assignment list --assignee c0409af9-7ccb-4e18-adf2-93dd50a39adc \
     --include-groups --include-inherited  \
     --scope "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog"

[
 {
   "condition": null,
   "conditionVersion": null,
   "createdBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
   "createdOn": "2023-09-23T07:49:44.104274+00:00",
   "delegatedManagedIdentityResourceId": null,
   "description": null,
   "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/441d2705-cbb4-40af-ad30-31d0f3422436",
   "name": "441d2705-cbb4-40af-ad30-31d0f3422436",
   "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
   "principalName": "My Subscription Group",
   "principalType": "Group",
   "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
   "roleDefinitionName": "Reader",
   "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
   "type": "Microsoft.Authorization/roleAssignments",
   "updatedBy": "51d2f4f3-b46e-4c80-a394-0fcd87449619",
   "updatedOn": "2023-09-23T07:49:44.104274+00:00"
 },
 {
   "condition": null,
   "conditionVersion": null,
   "createdBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
   "createdOn": "2024-03-20T12:55:28.371679+00:00",
   "delegatedManagedIdentityResourceId": null,
   "description": null,
   "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleAssignments/82daa968-312f-42e7-8934-7c596545607b",
   "name": "82daa968-312f-42e7-8934-7c596545607b",
   "principalId": "8bc9963a-1e72-4314-8703-3cd680368384",
   "principalName": "My Subscription Group",
   "principalType": "Group",
   "roleDefinitionId": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
   "roleDefinitionName": "Custom Contributor",
   "scope": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f",
   "type": "Microsoft.Authorization/roleAssignments",
   "updatedBy": "dde92205-34c0-4028-ba44-b2cd3d096701",
   "updatedOn": "2024-03-13T11:55:33.378901+00:00"
 }
]
  2. Describe the Custom Contributor role definition 
    ~/ ❯ az role definition list --name "Custom Contributor"

[
 {
   "assignableScopes": [
     "/subscriptions/6e85b02d-ec23-493e-bf2a-4be65492e6b5",
     "/subscriptions/96875dfe-c52a-4851-83b0-254863be206d",
     "/providers/Microsoft.Management/managementgroups/2d6bfa44-b523-4579-a70d-a145eaf272d5"
   ],
   "createdBy": "70b08e41-e31b-43db-8c57-b2ed5681d82c",
   "createdOn": "2019-07-24T12:59:19.599048+00:00",
   "description": "[Version 1.2] Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.",
   "id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/providers/Microsoft.Authorization/roleDefinitions/ee72b902-9895-4cbb-9a54-10ee567a1a52",
   "name": "ee72b902-9895-4cbb-9a54-10ee567a1a52",
   "permissions": [
     {
       "actions": [
         "*"
       ],
       "condition": null,
       "conditionVersion": null,
       "dataActions": [],
       "notActions": [
         "Microsoft.Authorization/*/Delete",
         "Microsoft.Authorization/*/Write",
         "Microsoft.Authorization/elevateAccess/Action",
         "Microsoft.Blueprint/blueprintAssignments/write",
         "Microsoft.Blueprint/blueprintAssignments/delete",
         "Microsoft.Compute/galleries/share/action",
         "Microsoft.Purview/consents/write",
         "Microsoft.Purview/consents/delete",
         "Microsoft.Subscription/rename/action",
         "Microsoft.Resources/deploymentStacks/manageDenySetting/action"
       ],
       "notDataActions": []
     }
   ],
   "roleName": "Custom Contributor",
   "roleType": "CustomRole",
   "type": "Microsoft.Authorization/roleDefinitions",
   "updatedBy": "366d6245-093c-4bc0-b8a0-dce37cdf383e",
   "updatedOn": "2024-05-16T08:53:05.530192+00:00"
 }
]
  3. Obtain AAD access token and set databricks host 
    ~/ ❯ export DATABRICKS_HOST="https://adb-1234567880123456.7.azuredatabricks.net"
~/ ❯ export DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq .accessToken -r)
  4. Demonstrate that the identity used has Metastore Admin permissions 
    ~/ ❯ databricks unity-catalog storage-credentials list

{
 "storage_credentials": [
   {
     "name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
     "azure_managed_identity": {
       "access_connector_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.Databricks/accessConnectors/unity-metastore-westeu",
       "managed_identity_id": "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/databricks-unity-metastore/providers/Microsoft.ManagedIdentity/userAssignedIdentities/unity-metastore-westeu",
       "credential_id": "c8aab434-60bc-44c0-a00a-84cf95b5c23c"
     },
     "read_only": false,
     "owner": "[email protected]",
     "id": "21ce434a-78c1-4a44-9850-c6ff2cceea6f",
     "metastore_id": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2",
     "created_at": 1696498690290,
     "created_by": "[email protected]",
     "updated_at": 1696498690290,
     "updated_by": "[email protected]",
     "used_for_managed_storage": true,
     "full_name": "ef6d050e-4555-4d8c-bab6-587ee1fea7f2-storage-credential-1234567890123",
     "securable_type": "STORAGE_CREDENTIAL",
     "securable_kind": "STORAGE_CREDENTIAL_AZURE_MI",
     "isolation_mode": "ISOLATION_MODE_OPEN"
   },
   ...
 ]
}
  5. Create storage credential using access connector ID and managed identity ID 
    ~/ ❯ databricks unity-catalog storage-credentials create --name test \
     --az-mi-access-connector-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog" \
     --az-mi-id "/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-connector-unity-catalog-mi"

     Traceback (most recent call last):
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 166, in perform_query
       resp.raise_for_status()
     File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
       raise HTTPError(http_error_msg, response=self)
   requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials

   During handling of the above exception, another exception occurred:

   Traceback (most recent call last):
     File "/usr/local/bin/databricks", line 8, in <module>
       sys.exit(cli())
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
       return self.main(*args, **kwargs)
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
       rv = self.invoke(ctx)
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
       return ctx.invoke(self.callback, **ctx.params)
     File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
       return __callback(*args, **kwargs)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 114, in wrapper
       f(*args, **kwargs)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/configure/config.py", line 67, in decorator
       return function(*args, **kwargs)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/cred_cli.py", line 166, in create_credential_cli
       cred_json = UnityCatalogApi(api_client).create_storage_credential(data, skip_val)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/api.py", line 103, in create_storage_credential
       return self.client.create_storage_credential(cred_spec, skip_validation)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/unity_catalog/uc_service.py", line 175, in create_storage_credential
       return self.client.perform_query('POST', url, data=cred_spec, headers=headers)
     File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 174, in perform_query
       raise requests.exceptions.HTTPError(message, response=e.response)
   requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://adb-1234567880123456.7.azuredatabricks.net/api/2.1/unity-catalog/storage-credentials
    Response from server:
    { 'details': [ { '@type': 'type.googleapis.com/google.rpc.ErrorInfo',
                    'domain': 'unity-catalog.databricks.com',
                    'metadata': { 'accessConnectorId': '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog'},
                    'reason': 'UC_STORAGE_CREDENTIAL_INVALID_CLOUD_PERMISSIONS'},
                  { '@type': 'type.googleapis.com/google.rpc.RequestInfo',
                    'request_id': 'eb684711-61ca-441b-ad34-8acc1e632b48',
                    'serving_data': ''}],
     'error_code': 'PERMISSION_DENIED',
     'message': 'Creating a storage credential requires the contributor role over '
                'the corresponding access connector with ID '
                '/subscriptions/c8a25cfd-bf10-4a03-9deb-eb99c90f251f/resourceGroups/my-resource-group/providers/Microsoft.Databricks/accessConnectors/my-connector-unity-catalog. '
                'Please contact your account admin.'}

Expected Behavior

Creation of the storage credential succeeded since the identity used has enough access permissions.

Actual Behavior

The attempt of a storage credential creation is failing with the following error: Creation of a storage credential required the contributor role over the corresponding access connector with ID '...'. Please contact your account admin.

OS and CLI version

~/ ❯ uname -s
Darwin

~/ ❯ echo $OSTYPE
darwin23.0

~/ ❯ databricks --version

Version 0.17.4

Is this a regression?

n.a.

Debug Logs

n.a.

mixam24 avatar Jun 10 '24 12:06 mixam24

Any comments on this one?

mixam24 avatar Jul 04 '24 08:07 mixam24

@mixam24 This sounds like a product feature/issue, not a CLI issue. I forwarded the question internally.

I doubt that it matters, but could you try doing the same with the new CLI? You're using the legacy Python one.

Installation instructions can be found at https://docs.databricks.com/en/dev-tools/cli/install.html.

pietern avatar Jul 04 '24 08:07 pietern

@pietern thanks for your reply.

This sounds like a product feature/issue, not a CLI issue.

I agree, the same problem arises in UI and in terraform (= anything that interacts with REST API).

I forwarded the question internally.

Could you kindly recommend an alternative communication channel for the discussion of this issue? I have already contacted Microsoft support, but I have serious doubts regarding their proficiency in the topic and overall effectiveness of the communication.

mixam24 avatar Jul 05 '24 10:07 mixam24

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

github-actions[bot] avatar Feb 05 '25 00:02 github-actions[bot]