dbt-databricks
dbt-databricks copied to clipboard
The tblproperties are not applied when using Python Model to create a table
Describe the bug
When using the python model, the tblproperties in config is not applied. The final result table does not contain those table properties.
Steps To Reproduce
Under the models/data.yml, i set the following configurations
config:
partition_by:
- event_date
tblproperties:
delta.compatibility.symlinkFormatManifest.enabled: false
delta.minReaderVersion: 2
delta.minWriterVersion: 7
delta.columnMapping.mode: name
delta.enableIcebergCompatV2: true
delta.universalFormat.enabledFormats: iceberg
In the models/data.py, it is
from pyspark.sql import functions as func
def model(dbt, session):
df = dbt.source("data", "accounts")
# do some transformation
return df
then I run dbt run --select data
Expected behavior
Those table properties should be respected.
Screenshots and log output
System information
The output of dbt --version:
Core:
- installed: 1.7.7
- latest: 1.7.7 - Up to date!
Plugins:
- databricks: 1.7.3 - Update available!
- spark: 1.7.1 - Up to date!
The operating system you're using:
Linux
The output of python --version:
Python 3.10.7
Additional context
I am using the Azure Databricks, DBR is 14.3 LTS.
Thanks for reporting.
This is somewhat fixed now, but since I don't yet have a PySpark API that lets me apply as part of create, it is applied as an alter, and I'm not sure that will achieve the outcome you're looking for here.