dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

Cannot materialize `view` models shared via Delta Sharing

Open shivonchain opened this issue 5 months ago • 4 comments

Describe the bug

When trying to materialize a view in dbt via dbt run or dbt build, and the view is shared via Delta Sharing, the materialization fails:

Runtime Error in model my_view_model (models/my_view_model.sql)
  [RequestId=xxxxx ErrorClass=INVALID_STATE] 
TABLE datacloud_prod_catalog.due_diligence_silver_views.my_view_model cannot be deleted 
because it is being shared via Delta Sharing. It is shared through the following shares: my_share. 
If you just want to update a shared view, please use ALTER VIEW instead.

To update a view definition when the underlying view is shared over Delta Sharing, ALTER VIEW should be used instead of CREATE OR REPLACE, which is issued by dbt for view models. Ideally, this is baked into the default view materialization in dbt-databricks.

An alternative I have considered is to use pre- and post-hooks to remove and add the view back to the underlying share. However, this is not always feasible.

Steps To Reproduce

In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.

  1. Create a view model in dbt.
  2. Share the view model with a data recipient via Delta Sharing.
  3. Run the view model using dbt run -s <model>.

Expected behavior

I expect the dbt view to be materialized even if it is shared by Delta Sharing. The view materialization should check if the view exists; if it does, then ALTER VIEW should be issued to Databricks instead of CREATE OR REPLACE to update the view definition.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

The output of dbt --version:

Core:
  - installed: 1.8.2
  - latest:    1.8.6 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.8.1 - Update available!
  - spark:      1.8.0 - Up to date!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

The operating system you're using: macOS, M3 Max

The output of python --version: Python 3.12.3

Additional context

Add any other context about the problem here.

shivonchain avatar Aug 30 '24 11:08 shivonchain