Support Delta Live Tables Expectations

[stevenayers-bge]

Describe the feature

Add support for DLT Expectations: Manage data quality with Delta Live Tables

These could utilize the constraints resource property for syntax:

models:
  - name: lendingclub_clean
    materialized: streaming_table
    constraints:
      - name: expectation_1
        type: expectation
        expression: (avg_cur_bal >= 0) ON VIOLATION DROP ROW
    columns:
      - name: avg_cur_bal
        data_type: int

becomes:

CREATE LIVE TABLE lendingclub_clean(
  CONSTRAINT expectation_1 EXPECT (avg_cur_bal >= 0) ON VIOLATION DROP ROW
)

Describe alternatives you've considered

I use normal tables at the moment with ordinary DBT tests, but as the ecosystem grows around DLT it's starting to provide some critical Data Quality capabilities such as Lakehouse Monitoring which I'd love to be able to take advantage of.

Additional context

If this is already supported, please let me know. I couldn't find anything on the topic.

Who will this benefit?

It would give people who are using DBT on Databricks a huge incentive to start using DLT.

Are you interested in contributing this feature?

Yes

[benc-db]

It's not supported yet, but it's something we're strongly considering to add this year. Thanks for filing the ticket.

[stevenayers-bge]

It's not supported yet, but it's something we're strongly considering to add this year. Thanks for filing the ticket.

Thanks @benc-db. Is APPLY CHANGES INTO supported yet? Same again for that, happy to contribute if it isn't or hasn't been started yet.

[benc-db]

This is the first that APPLY CHANGES INTO is on my radar; will share with my managers. If you'd like to contribute to anything MV/ST related, make sure you start from 2.0.latest (release coming in May), as that has significant changes to MV/ST stuff in it.