sdk
sdk copied to clipboard
feat: Bulk-aliasing of stream properties with respect to property name using `stream_maps`
Feature scope
Taps (catalog, state, tests, etc.)
Description
If I have a bunch of stream properties I want to rename, currently I have to follow the "copy-and-delete" approach as per aliasing properties:
stream_maps:
users:
firstName: first_name
lastName: last_name
first_name: __NULL__
last_name: __NULL__
It would be nice to encapsule this into some property-level aliasing construct along with a new scoped __property_name__ variable (adaptation of https://github.com/meltano/sdk/issues/2490#issuecomment-2642809885)
stream_maps:
users:
__map__: '"".join("_" + char.lower() if char.isupper() else char for char in __property_name__)' # example converting camelCase to snake_case
__map_filter__: # filter specific properties to map with glob expression support; default to all if omitted
- *_name
or this proposal from https://github.com/meltano/sdk/issues/2545#issuecomment-2234055006
config: stream_maps: {STREAM_NAME}: __meta__[col['name'].lower()]: col
This makes sense to me (the __map__ proposal).
We have to keep in mind that the stream map definition must also affect the generated SCHEMA messages. I can't see any complications with that at the moment, just a word of caution 😅
This came up again today in Slack and got me thinking whether or not a wildcard pattern approach would work?
stream_maps:
users:
'*Name': '"".join("_" + char.lower() if char.isupper() else char for char in __property_name__)'
I think there would be some complexity in making the distinction between a mapping for keys vs values here though.