snowflake-kafka-connector icon indicating copy to clipboard operation
snowflake-kafka-connector copied to clipboard

Published 20 hours ago • verified publisher icon snowflakedb

UPSERT Support

Open miramar-labs opened this issue 2 years ago • 4 comments

I'm using the JDBC Source connector to bring a MSSQL table into kafka as a topic. I have two sink connectors configured on that topic .. the JDBC Sink connector (pushing the data back out to an identical table in MSSQL) and the Snowflake Sink Connector (where I'm trying to do the same, but as a Snowflake table). The JDBC sink seems to handle upserts just fine, in that say the source table contains 3 rows and I update one of them, the destination table contains exactly 3 rows with one updated. The snowflake sink does not seem to work this way ... updates appear in the destination table as new rows (with the same key but different timestamps). Q: How can I do proper UPSERT to a Snowflake table? Or is this not supported?

miramar-labs avatar Oct 26 '22 18:10 miramar-labs

I have also tried creating a stream on the raw VARIANT snowflake table, with the idea that I could use a task to periodically run a MERGE into a target table, but no matter if I INSERT or UPDATE rows in the source database, the stream always shows them as INSERTS.. so the merge won't work in that case.

miramar-labs avatar Oct 31 '22 17:10 miramar-labs

Any plans to support update/upsert functionality in the near future? I'm curious how that plays with the snow pipe streaming?

lyuboxa avatar Jan 16 '24 20:01 lyuboxa