pipewrench
pipewrench copied to clipboard
Examples and templates around 'metadata' properties is not fully updated.
Before, specific metadata tags such as META_SOURCE
were added to a table. A later addition added a metadata
tag so arbitrary fields could be treated as metadata and added to hive table properties
table:
metadata:
source: some relational db
contactInfo: [email protected]
Some templates still use the old metadata format: https://github.com/Cargill/pipewrench/blob/d0b06fbe1876481ace14f48f8adee45563193b2d/templates/sqoop-parquet-hdfs-hive-merge/create-report-table.sql
This example will probably run with the new format but still has the old tags: https://github.com/Cargill/pipewrench/blob/422f396baa726c01f78a7dcb3dc1aa98182083a1/examples/sqoop-parquet-full-load/tables.yml
All the 'old' style metadata should be removed.
I am working on it Tony. @afoerster