sdk icon indicating copy to clipboard operation
sdk copied to clipboard

feat: Bulk-aliasing of stream properties with respect to property name using `stream_maps`

Open ReubenFrankel opened this issue 8 months ago • 2 comments

Feature scope

Taps (catalog, state, tests, etc.)

Description

If I have a bunch of stream properties I want to rename, currently I have to follow the "copy-and-delete" approach as per aliasing properties:

stream_maps:
  users:
    firstName: first_name
    lastName: last_name
    first_name: __NULL__
    last_name: __NULL__

It would be nice to encapsule this into some property-level aliasing construct along with a new scoped __property_name__ variable (adaptation of https://github.com/meltano/sdk/issues/2490#issuecomment-2642809885)

stream_maps:
  users:
    __map__: '"".join("_" + char.lower() if char.isupper() else char for char in __property_name__)'  # example converting camelCase to snake_case
    __map_filter__:  # filter specific properties to map with glob expression support; default to all if omitted
    - *_name

or this proposal from https://github.com/meltano/sdk/issues/2545#issuecomment-2234055006

    config:
       stream_maps:
         {STREAM_NAME}:
           __meta__[col['name'].lower()]: col

ReubenFrankel avatar Mar 10 '25 12:03 ReubenFrankel

This makes sense to me (the __map__ proposal).

We have to keep in mind that the stream map definition must also affect the generated SCHEMA messages. I can't see any complications with that at the moment, just a word of caution 😅

edgarrmondragon avatar Mar 12 '25 22:03 edgarrmondragon

This came up again today in Slack and got me thinking whether or not a wildcard pattern approach would work?

stream_maps:
  users:
    '*Name': '"".join("_" + char.lower() if char.isupper() else char for char in __property_name__)'

I think there would be some complexity in making the distinction between a mapping for keys vs values here though.

ReubenFrankel avatar Sep 04 '25 13:09 ReubenFrankel