pipewrench icon indicating copy to clipboard operation
pipewrench copied to clipboard

Published 20 hours ago • verified publisher icon Cargill

airflow support for pipewrench

Open SandishKumarHN opened this issue 6 years ago • 3 comments

airflow support for pipewrench

SandishKumarHN avatar Mar 22 '18 15:03 SandishKumarHN

@SandishKumarHN this will be a brand new set of templates?

afoerster avatar Mar 22 '18 17:03 afoerster

Yes, in airflow task-id would be like this example-> sqoop_create_exec_{{ conf.source_database.name }}_{{ table.source.name }} and filenames would be same but .py or we can do like examples-> create_execute_sqoop_parquet_hdfs_impala.py and clean_sqoop_parquet_hdfs_impala.py

SandishKumarHN avatar Mar 22 '18 17:03 SandishKumarHN

I'm not sure I'm following, share a WIP branch when you get a rough outline.

afoerster avatar Mar 22 '18 18:03 afoerster