sdk icon indicating copy to clipboard operation
sdk copied to clipboard

Feat: add the `json` module to stream maps context

Open aaronsteers opened this issue 2 years ago • 5 comments
trafficstars

Per

  • #1500

Specifically, json.loads() and json.dumps() would be helpful.

aaronsteers avatar Mar 16 '23 00:03 aaronsteers

@aaronsteers Do we need all members of json or can we do with just json.loads and json.dumps?

edgarrmondragon avatar Mar 16 '23 00:03 edgarrmondragon

@aaronsteers Do we need all members of json or can we do with just json.loads and json.dumps?

Good question. Want to add thoughts in the PR?

I like adding json because it's simplest and doesn't require retraining from what users would expect. But I don't mind if we want to implement in another way - especially if for security reasons we want to keep to "safe" implementations only.

Note:

For md5, there just was no good inline way to calculate hashes, so we had to create our own custom function. With json library, the inline method works, so we don't strictly need to do as much in this case.

aaronsteers avatar Mar 16 '23 00:03 aaronsteers

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

stale[bot] avatar Jul 18 '23 03:07 stale[bot]

Still relevant

edgarrmondragon avatar Jul 20 '23 22:07 edgarrmondragon

Things to keep in mind for anyone that's interested in contributing this:

  • How to communicate schema types? Perhaps aliases of json.loads should be used, which the SDK can identify as specific types, e.g. parse_json_object, parse_json_array, etc.
  • See https://github.com/meltano/sdk/blob/97d97de313511be4c21b842bf2ba0283d4bf5c02/singer_sdk/mapper.py#L297-L307

edgarrmondragon avatar Sep 15 '23 16:09 edgarrmondragon