dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

The tblproperties are not applied when using Python Model to create a table

Open csimplestring opened this issue 1 year ago • 2 comments
trafficstars

Describe the bug

When using the python model, the tblproperties in config is not applied. The final result table does not contain those table properties.

Steps To Reproduce

Under the models/data.yml, i set the following configurations

config:
    partition_by:
    - event_date
    tblproperties:
        delta.compatibility.symlinkFormatManifest.enabled: false
        delta.minReaderVersion: 2
        delta.minWriterVersion: 7
        delta.columnMapping.mode: name
        delta.enableIcebergCompatV2: true
        delta.universalFormat.enabledFormats: iceberg

In the models/data.py, it is

from pyspark.sql import functions as func

def model(dbt, session):

    df = dbt.source("data", "accounts")
    # do some transformation
    return df

then I run dbt run --select data

Expected behavior

Those table properties should be respected.

Screenshots and log output

Screenshot 2024-02-14 at 11 29 44 As you can see, only the default table properties are used, the new properties are not used at all.

System information

The output of dbt --version:

Core:
  - installed: 1.7.7
  - latest:    1.7.7 - Up to date!

Plugins:
  - databricks: 1.7.3 - Update available!
  - spark:      1.7.1 - Up to date!

The operating system you're using: Linux The output of python --version: Python 3.10.7

Additional context

I am using the Azure Databricks, DBR is 14.3 LTS.

csimplestring avatar Feb 14 '24 10:02 csimplestring

Thanks for reporting.

benc-db avatar Feb 14 '24 17:02 benc-db

This is somewhat fixed now, but since I don't yet have a PySpark API that lets me apply as part of create, it is applied as an alter, and I'm not sure that will achieve the outcome you're looking for here.

benc-db avatar May 09 '24 22:05 benc-db