dbt-databricks
dbt-databricks copied to clipboard
support user incremental predicates on merge
Describe the feature
Users should be able to supply incremental models with arbitrary predicates during the merge step to prevent unnecessary table scan for massive data sets.
Describe alternatives you've considered
None
Additional context
See:
Who will this benefit?
users with extremely large datasets who want to limit the scan of their merge operations on incremental models!
Are you interested in contributing this feature?
Yes
@ueshin @superdupershant seems like a reasonable ask since it's supported in other DWs.