snowflake-kafka-connector icon indicating copy to clipboard operation
snowflake-kafka-connector copied to clipboard

Document prerequisites to enable exactly-once delivery guarantees

Open chrisVia opened this issue 2 years ago • 1 comments

I'm trying to run integration-tests on this project (release 1.8) to validate some changes I've added to a fork. It seems I am unable to run any test which tries to create a "client-sequencer" from the new snowflake-ingest-sdk (this code-path seems to be generally reached when one configures the connector for exactly-once delivery guarantees). The ingest-sdk reports the following unauthorization error to me:

Message: Not authorized to manage the specified object. Snowpipe APIs used to configure client, get status and insertFiles with clientInfo are disabled,

Can you tell me what I need to do to my snowflake user to enable these API endpoints so I can run all the integration tests? Thanks!

chrisVia avatar Jun 24 '22 22:06 chrisVia

Looks like it's because the EOS feature is still in PrPr, @sfc-gh-japatel any idea about a workaround here?

sfc-gh-tzhang avatar Jul 14 '22 00:07 sfc-gh-tzhang

There is no workaround here apart from enabling the new APIs on server side for their account. As of today:

  • We are not planning to get this feature out of PrPr.
  • We will discontinue the use of this config. (1.7.x wont be recommended to use), hence removal of this tests.
  • We are in PuPr for our Snowpipe Streaming APIs which gives all benefits of EXACTLY_ONCE feature we announced with snowpipe last year.

Closing this issue. @chrisVia feel free to reopen again.

sfc-gh-japatel avatar Mar 10 '23 19:03 sfc-gh-japatel