Support arbitrary key-value pairs in metadata

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/320

Originally created by @edgarrmondragon on 2022-01-29 03:25:39

---

Summary

According to the singer metadata spec:

A tap is free to write ANY type of metadata they feel is useful for describing fields in the schema, although several reserved keywords exist.

However, our dataclass implementation prevents the addition of arbitrary fields to the metadata by the tap developer. So, something like this is not possible:

{
  "metadata" : {
    "selected" : true,
    "some-other-metadata" : "whatever"
  },
  "breadcrumb" : ["properties", "some-field-name"]
}

Proposed benefits

Taps can emit informational metadata. This tags can be later leveraged by data catalogs and orchestrators (https://gitlab.com/meltano/meltano/-/issues/2611+).

Proposal details

As suggested above, tap devs interested in adding key-value pairs to their metadata objects would have to subclass _singer.Metadata etc. This is complicated by the use those of classes in Tap and Stream.

To avoid that, we could easily add a tags property that tap developers can fill-in with relevant metadata:

{
  "metadata" : {
    "selected" : true,
    "tags": {
      "some-other-metadata" : "whatever"
    }
  },
  "breadcrumb" : ["properties", "some-field-name"]
}

Best reasons not to build

There may be better ways to support the addition of extra metadata fields.

[MeltyBot]

View 2 previous comments from the original issue on GitLab

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still open and https://github.com/meltano/sdk/labels/Accepting%20Pull%20Requests

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.