snowflake-kafka-connector icon indicating copy to clipboard operation
snowflake-kafka-connector copied to clipboard

[Feature] RecordNameStrategy handling

Open HenkvanDyk opened this issue 4 months ago • 3 comments

Hi team,

We're using recordNameStrategy for schemas on certain topics, meaning topics with differing content arrives on the same Kafka topic. While trying to split out this data once it arrives on Snowflake it become quite hard to infer which schema is used for certain messages.

Is it possible to have the schemaid or subject name used during Avro deserialization passed in the record_metadata to the snowflake tables created? I believe this would require some modification on how the AvroConverter returns messages, or is this something that is being returned but not used by the Snowflake connector?

Another option would be to add a mechanism to dynamically create snowflake table names based on the subject of the schema, would something like this be possible?

Otherwise, any recommendations on how to handle this?

HenkvanDyk avatar Oct 10 '24 11:10 HenkvanDyk