connect
connect copied to clipboard
redpanda-data
Snowflake SQL Driver Option
Is there any interest in having Snowflake as a driver option in the SQL processor/output? The driver can be found here: https://github.com/snowflakedb/gosnowflake
Hey @nmbryant yes absolutely, any idea if there's a way to test it without using a snowflake account?
Unfortunately it doesn't seem like testing can be without an account from what I have found. I think adding the driver option would be a small change, should I just make the change without tests?
Snowflake is pretty good with open source folks from what I've heard. May be worth a shout out to someone like mpilman. If not, I have access to snowflake and have used benthos in the past ... if I can swing some cycles I could potentially help.
I'm exploring the library right now and trying to build an output. @ironchef do you have experience with the Snowflake platform? I'd love to have a chat with someone who can provide some feedback on how people expect this to work in practice, especially from a data streaming perspective.
Yup yup .. mostly from a "hey .. execute this query and give me results" perspective on the golang side. But I've integrated snowflake with AWS + S3, airflow, dbt, etc. (a bunch of other tooling). I'll dm / email ya.
@ironchef I created a PR for adding a snowflake_put output in #1160. If you have some time to try it out, I'd really appreciate some feedback.
Potentially, we can also have a snowflake_insert output, but there are some caveats to that as described in this issue I opened on the gosnowflake client repo: https://github.com/snowflakedb/gosnowflake/issues/540
Update: The snowflake_put output was merged in #1160. I think it's worth adding a corresponding input and, potentially, some alternatives based on SELECT / INSERT. Leaving this open for feedback.
Closing this since, besides the existing snowflake_put output, Snowflake support was added to the various sql_* components in v4.10.0.