azure-kusto-spark icon indicating copy to clipboard operation
azure-kusto-spark copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Move extents failover

Open ohadbitt opened this issue 3 years ago • 4 comments

Move extents failure of one batch - should revert change if its a partial success or increase the number of tries after first success

ohadbitt avatar Aug 10 '21 06:08 ohadbitt

We use spark driver to insert data in exactly once semantics: we mark data batches with unique tags on ingest and add ingestIfNotExists in ingestion properties. There is a problem in KustoClient finalizeIngestionWhenWorkersSucceeded method. If spark cluster fails, we can stay in an intermediate state, some extents with our unique tag will be ingested, but some will not because of batching strategy in this method. I suppose problem with a lot of extents ingestion can be solved only by cluster scale out.

k-tomak avatar Dec 14 '21 08:12 k-tomak

Do you have any news about this request? We can not upgrade to the next version of library because it isn't safety.

k-tomak avatar Mar 05 '22 10:03 k-tomak

This actually shouldnt concern most people - with newer version - with retries implemented and the ability to set the option KUSTO_MAXIMAL_EXTENTS_COUNT_FOR_SPLIT_MERGE_PER_NODE "maximalExtentsCountForSplitMergePerNode" this shouldn't be a problem.

ohadbitt avatar Aug 21 '22 11:08 ohadbitt

@ohadbitt since are now attempting retries in move extents as well, should we go ahead and close this task

ag-ramachandran avatar Dec 19 '22 11:12 ag-ramachandran