snowflake-ingest-java icon indicating copy to clipboard operation
snowflake-ingest-java copied to clipboard

Published 20 hours ago • verified publisher icon snowflakedb

SNOW-1512047 Introduce independent per-table flushes when interleaving is disabled

Open sfc-gh-alhuang opened this issue 7 months ago • 0 comments

The current SDK behavior flushes all channels simultaneously when any buffer reaches its limit, potentially causing unnecessary small file flushes if interleaving is disabled (MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST = 1) and the ingestion throughput between tables is uneven.

As the WIP streaming to Iceberg table feature set MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST = 1. This PR introduces per-table flushing to avoid the above issue.

JIRA

sfc-gh-alhuang avatar Jul 03 '24 22:07 sfc-gh-alhuang