Consider a reusable data transformation framework

[JJediny]

Really great project!

Scanning the code, noticed the ETL/transformation/mapping portions are fairly hardcoded to the data source (which could make it hard to maintain if their schemas' change). Just sharing some projects that might be worth considering as a way to provide reusable schema translation/templating - from the various data sources you all are tracking in issues:

graphql native <~ variety of source APIs

• https://github.com/Urigo/graphql-mesh

JSON <~ REST using JMESpath

• https://github.com/JJediny/json-schema-defined-mapper
• https://github.com/caljrimmer/1liner
• https://github.com/Comcast/eel
• JSONpath or jq as alternatives to JMESpath

For Reference Only

• Not suggesting this as a technical solution, but provides the context behind this issue: https://docs.confluent.io/platform/current/schema-registry/index.html