snowflake-kafka-connector icon indicating copy to clipboard operation
snowflake-kafka-connector copied to clipboard

Support multiple topics to one table in snowflake.topic2table.map

Open globalpay-wafishe opened this issue 4 years ago • 6 comments

Currently, part of the validation of the snowflake.topic2table.map parameter requires all of the tables to be unique so multiple topics cannot be written to the same table

"topics": "topic1,topic2",
"snowflake.topic2table.map": "topic1:table1,topic2:table2",

We have multiple topics with the same data format though, and would like to be able to write them to the same table to reduce the special processing we need to do on the snowflake side

"topics": "topic1,topic2",
"snowflake.topic2table.map": "topic1:table1,topic2:table1",

globalpay-wafishe avatar Jan 05 '21 14:01 globalpay-wafishe

I built a JAR with the check stopping this removed and ran it for about a day and the combined table got the same data as the three individual ones union'd together. I would like to make a PR for my branch but it doesn't look like I have permission to push it to origin.

globalpay-wafishe avatar Jan 13 '21 12:01 globalpay-wafishe

Looks like no movement on this? This should definitely be supported.

jamesgregorystewart avatar Sep 21 '21 18:09 jamesgregorystewart

I'd also like to use this feature.

koenigroland avatar Sep 22 '21 15:09 koenigroland

Has there been any update on this feature?

+1 for wanting support for this feature

Dormira avatar Apr 07 '22 19:04 Dormira

+1

grzegorzb1990 avatar Jun 14 '22 11:06 grzegorzb1990

This should be supported now as part of 1.8.1 release, please give it a try!

sfc-gh-tzhang avatar Sep 12 '22 20:09 sfc-gh-tzhang