public-datasets-pipelines
public-datasets-pipelines copied to clipboard
Templating tool for simple pipelines
Description
Instead of going through the pipeline development instructions for simple pipelines (such as a simple CSV to BigQuery dump), we can use a templating tool that creates all the necessary files and folders, with the pipeline config being auto-populated based on some set of inputs.
For example,
$ create_pipeline_template \
--pipeline-group "yadda" \
--pipeline-name "my_simple_pipeline" \
--source="https://some/data.csv" \
--destination-type "bigquery" \
--destination-name "dataset.some_bigquery_table" \
--rename-column "Number of Cases:cases" \
--rename-column "Date:date" \
--remove-null "column=date"
Checklist
- [x] I created this issue in accordance with the Code of Conduct.
- [x] This issue is appropriately labeled.