dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

support user incremental predicates on merge

Open dave-connors-3 opened this issue 2 years ago • 1 comments

Describe the feature

Users should be able to supply incremental models with arbitrary predicates during the merge step to prevent unnecessary table scan for massive data sets.

Describe alternatives you've considered

None

Additional context

See:

Who will this benefit?

users with extremely large datasets who want to limit the scan of their merge operations on incremental models!

Are you interested in contributing this feature?

Yes

dave-connors-3 avatar Aug 23 '22 14:08 dave-connors-3

@ueshin @superdupershant seems like a reasonable ask since it's supported in other DWs.

bilalaslamseattle avatar Aug 26 '22 12:08 bilalaslamseattle