snowflake-ingest-java icon indicating copy to clipboard operation
snowflake-ingest-java copied to clipboard

Published 20 hours ago • verified publisher icon snowflakedb

SNOW-1437884 Flush service supports uploading file to Iceberg tables

Open sfc-gh-alhuang opened this issue 8 months ago • 0 comments

Modify flush service to support uploading file to external volume. This PR includes following change:

  • Add default parameter MAX_CLIENT_LAG for Iceberg mode
  • Introduce new parameter DISABLE_CHUNK_ENCRYPTION to disable chunk encryption in Iceberg mode
  • Move stage from client level to table level when using Iceberg mode, see: StreamingIngestExternalVolume.java
  • Move stage/volume configure logic to channel level when using Iceberg
  • Fix FlushServiceTest and StreamingIngestStage, more test cases is expected

sfc-gh-alhuang avatar Jun 08 '24 01:06 sfc-gh-alhuang