pipewrench icon indicating copy to clipboard operation
pipewrench copied to clipboard

Examples and templates around 'metadata' properties is not fully updated.

Open afoerster opened this issue 6 years ago • 1 comments

Before, specific metadata tags such as META_SOURCE were added to a table. A later addition added a metadata tag so arbitrary fields could be treated as metadata and added to hive table properties

table:
  metadata:
     source: some relational db
     contactInfo: [email protected]

Some templates still use the old metadata format: https://github.com/Cargill/pipewrench/blob/d0b06fbe1876481ace14f48f8adee45563193b2d/templates/sqoop-parquet-hdfs-hive-merge/create-report-table.sql

This example will probably run with the new format but still has the old tags: https://github.com/Cargill/pipewrench/blob/422f396baa726c01f78a7dcb3dc1aa98182083a1/examples/sqoop-parquet-full-load/tables.yml

All the 'old' style metadata should be removed.

afoerster avatar Aug 09 '18 14:08 afoerster

I am working on it Tony. @afoerster

sada3390 avatar Oct 26 '18 18:10 sada3390