dig-etl-engine
dig-etl-engine copied to clipboard
Add ability to combine columns using templates
There are CSV files where the values are split over multiple columns and we need them as one column. For example, there can be year, month and day columns and we need a date attribute.
Use a feature similar to the template scheme used for titles to combine multiple columns. The syntax can be:
{
"path": "date",
"field": "event_date",
"template": "{year}/{month}/{day}"
}
When the template option is present, the path contains the name of a new attribute for the output JSON, i.e., in the above example, date is not a column in the CSV, but will be an attribute in the JSON. The value of date is computed by the template based on other columns in the CSV.
Added template feature to combine fields.
Not closing because we may need something smarter for the cases when some fields are missing, eg, sometimes we are supposed to have year, month and day, but day is missing so a simple template creates illegal dates.
WIll keep this open, it should be fixed in etk2