airflow-clickhouse-plugin icon indicating copy to clipboard operation
airflow-clickhouse-plugin copied to clipboard

Add template_searchpath support

Open tiger24l7 opened this issue 3 years ago • 2 comments

Hi! The problem is that ClickhouseOperator doesn't support template_searchpath dag option which allows to pass filename instead of full query

Instead of:

ClickHouseOperator(
    task_id='task_1',
    clickhouse_conn_id='clickhouse_default',
    sql='SELECT * from db.some_table;' # <-- raw SQL query
)

We can write:

ClickHouseOperator(
    task_id='task_1',
    clickhouse_conn_id='clickhouse_default',
    sql='select_query.sql' # <-- filename of jinja template which is located in template_searchpath location
)

One more example how it works

tiger24l7 avatar Oct 24 '22 10:10 tiger24l7

Look like it's already done https://github.com/whisklabs/airflow-clickhouse-plugin/pull/41 But requires some additional things

tiger24l7 avatar Oct 24 '22 10:10 tiger24l7

Hi @tiger24l7, thank you for you feedback. Yes, there is an open PR, you are welcome to contribute :)

bryzgaloff avatar Oct 25 '22 03:10 bryzgaloff

My python skills are poor I might try to implement this later But currently, would be nice if someone implement this, if possible :)

tiger24l7 avatar Dec 01 '22 11:12 tiger24l7

Contribution is welcome, thank you! I will review you PR and let you know how you may enhance the code.

bryzgaloff avatar Dec 01 '22 11:12 bryzgaloff

Implemented in #62

bryzgaloff avatar Jan 16 '23 10:01 bryzgaloff