elasticsearch-net icon indicating copy to clipboard operation
elasticsearch-net copied to clipboard

Add overload to BulkAll with IAsyncEnumerable

Open nesterenko-kv opened this issue 4 years ago • 2 comments

It would be great to see support for IAsyncEnumerable, because, when pulling data from the database asynchronously, it requires to load data completely into memory in order to use synchronous enumeration (IEnumerable),

nesterenko-kv avatar Aug 14 '20 23:08 nesterenko-kv

Hi @nesterenko-kv

when pulling data from the database asynchronously, it requires to load data completely into memory in order to use synchronous enumeration (IEnumerable),

The IEnumerable<T> implementation passed to BulkAll does not explicitly require loading data completely into memory. For example, it might be an implementation that executes queries against the database to fetch just a page of results at a time.

Were you referring to the loading of each page completely into memory (which currently happens)? There might be some benefit to changing this to support IAsyncEnumerable<T>, though I think it would also require extending how bulk requests are serialized to the request stream to take full advantage, since bulk operations are currently buffered on the request.

russcam avatar Aug 17 '20 02:08 russcam