Filter Columns from CSV Sources

[tiagotalbuquerque]

Would be nice to have something to filter undesired columns from CSV source. It can avoid a lot of hassle from CSV formating when we are importing data.

[bhou]

Currently, it is possible to filter the undesired columns by an SQL step with

SELECT [desired columns] FROM PCOLLECTION

It is a whitelist way. But sometimes, it is also useful to 'blacklist' the undesired column. This could be done in CSV Source, for example, a config

config:
    undesiredColumns:
        - field1
        - field2