public-datasets-pipelines icon indicating copy to clipboard operation
public-datasets-pipelines copied to clipboard

Templating tool for simple pipelines

Open adlersantos opened this issue 4 years ago • 0 comments

Description

Instead of going through the pipeline development instructions for simple pipelines (such as a simple CSV to BigQuery dump), we can use a templating tool that creates all the necessary files and folders, with the pipeline config being auto-populated based on some set of inputs.

For example,

$ create_pipeline_template \
    --pipeline-group "yadda" \
    --pipeline-name "my_simple_pipeline" \
    --source="https://some/data.csv" \
    --destination-type "bigquery" \
    --destination-name "dataset.some_bigquery_table" \
    --rename-column "Number of Cases:cases" \
    --rename-column "Date:date" \
    --remove-null "column=date"

Checklist

  • [x] I created this issue in accordance with the Code of Conduct.
  • [x] This issue is appropriately labeled.

adlersantos avatar May 20 '21 15:05 adlersantos