azure-kusto-spark icon indicating copy to clipboard operation
azure-kusto-spark copied to clipboard

Feature/add kusto streaming feature

Open ag-ramachandran opened this issue 1 year ago • 1 comments

Pull Request Description

This is the PR that was for Kusto Streaming. Cherry-picked from the PR #301 and rebased with master

This adds a new option called Stream, that streams data using ManagedStreamingIngest client to Kusto.

Did a couple of large runs to Kusto . While it seems to run satisfactorily, many times it falls back to queued ingest in case the volumes are high

image

Lower volumes seem to be using streaming quite effectively

image

Future Release Comment

[Add description of your change, to include in the next release] [Delete any or all irrelevant sections, e.g. if your change does not warrant a release comment at all]

Breaking Changes:

  • None

Features:

  • None

Fixes:

  • None

ag-ramachandran avatar Feb 05 '24 15:02 ag-ramachandran

Test Results

77 tests  +2   73 :white_check_mark:  - 2   2m 17s :stopwatch: -25s 22 suites ±0    0 :zzz: ±0  22 files   ±0    4 :x: +4 

For more details on these failures, see this check.

Results for commit 82952325. ± Comparison against base commit 2750c0eb.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Feb 28 '24 06:02 github-actions[bot]