sdk
sdk copied to clipboard
Ability to inherit data classification rules from upstream column metadata and/or column naming patterns
trafficstars
This would allow the data classification to be added by the tap developer or the tap user, which in turn would pass downstream to the mapper and also the target, each of which could leverage this when deciding whether to hash and how/whether to serialize in the downstream data source.
This capability could be delivered in 3 (or more?) ways which could optionally be split into separate features or issues:
- For SQL sources: if the source table supports column comments or metadata, our SQL SDK handling could get that column metadata and add data classification to the tap's stream.
- For non-SQL sources like Salesforce, we could provide a pre-defined interface for tap developers to implement the same.
- As a general application for all SDK-based taps, we can provide a config option for users to explicitly apply certain data classification labels to source properties - based on column name or column naming pattern.
Related to, and pairs with:
- #1537