dig-etl-engine icon indicating copy to clipboard operation
dig-etl-engine copied to clipboard

Add ability to combine columns using templates

Open szeke opened this issue 7 years ago • 2 comments

There are CSV files where the values are split over multiple columns and we need them as one column. For example, there can be year, month and day columns and we need a date attribute.

Use a feature similar to the template scheme used for titles to combine multiple columns. The syntax can be:

{
            "path": "date",
            "field": "event_date",
            "template": "{year}/{month}/{day}"
}

When the template option is present, the path contains the name of a new attribute for the output JSON, i.e., in the above example, date is not a column in the CSV, but will be an attribute in the JSON. The value of date is computed by the template based on other columns in the CSV.

szeke avatar Jan 07 '18 17:01 szeke

Added template feature to combine fields.

Not closing because we may need something smarter for the cases when some fields are missing, eg, sometimes we are supposed to have year, month and day, but day is missing so a simple template creates illegal dates.

szeke avatar Jan 16 '18 07:01 szeke

WIll keep this open, it should be fixed in etk2

saggu avatar Jul 11 '18 22:07 saggu