dbt-sqlserver icon indicating copy to clipboard operation
dbt-sqlserver copied to clipboard

support Ephemeral models

Open dataders opened this issue 4 years ago • 1 comments

from #137, here's the proposed solution

Ephemeral models

The general approach outlined in https://github.com/dbt-msft/dbt-synapse/commit/9180da5a6d092c6b4b22d5388aa6592d2b9d77c4 is still the same, though I was doing it on easy mode, and didn't actually try to solve for ephemeral model compilation :)

The mechanism is: reimplement the compiler, define a new one (like in dbt/adapters/sqlserver/compilation.py), and reimplement methods like _inject_ctes_into_sql and _recursively_prepend_ctes.

Taking a step back: In order to support ephemeral models with CTEs on SQLServer and Synapse, given that T-SQL doesn't support CTEs nested inside either CTEs or subqueries, would we have to take the same approach as the one outlined above?

Namely, instead of recursively prepending ephemeral models as CTEs to the start of the query:

with  __dbt__cte__ephemeral_model as (


with my_cte as (
    
    select 1 as id
    
)

select * from my_cte
),ephemeral_model as (

select * from __dbt__cte__ephemeral_model

),

another_cte as (
    
    select 2 as id
    
)

select * from ephemeral_model
union all
select * from another_cte

Recursively prepend them as temp tables, to be executed in-transaction with the body of the query:

create table #ephemeral_model as (


with my_cte as (
    
    select 1 as id
    
)

select * from my_cte
);

with ephemeral_model as (

select * from #ephemeral_model

),

another_cte as (
    
    select 2 as id
    
)

select * from ephemeral_model
union all
select * from another_cte

dataders avatar Oct 14 '21 18:10 dataders

I don't think we should do this :)

Newer MSFT databases should support nested CTEs (https://github.com/dbt-msft/dbt-sqlserver/issues/457). We shouldn't expose the full Compiler interface to adapters (https://github.com/dbt-labs/dbt-core/pull/9134).

jtcohen6 avatar Nov 28 '23 10:11 jtcohen6