astronomer-cosmos icon indicating copy to clipboard operation
astronomer-cosmos copied to clipboard

[Feature] Emptytask for ephemeral materializations

Open w0ut0 opened this issue 6 months ago • 0 comments

Description

I would like the ability (or have it built-in) that ephemeral materialized tasks are EmptyOperators (or something similar).

Use case/motivation

Running an ephemeral model does not affect the database (I assume?).

However, a whole DbtRunOperator runs for these tasks, which need their own parsing,.. and might take a couple of seconds. The task also launches some queries to the database (list tables,..) which are unnecessary. One of our projects makes heavy use of ephemeral models, and this both clutters the DAG , and takes up valuable worker time.

I tried excluding them in the RenderConfig, which works, but I lose all the dependencies/refs that go 'through' this model, leading my DAG to run out-of-order.

The node_converters would be nice for this, however you cannot apply a different converter depending on materialization (I think?).

Related issues

This issue is related, however simply not rendering the models leads to missing refs (ideally these would be inferred and the ephemeral models would not be rendered.

Are you willing to submit a PR?

  • [ ] Yes, I am willing to submit a PR!

w0ut0 avatar Aug 01 '24 08:08 w0ut0