conveyor icon indicating copy to clipboard operation
conveyor copied to clipboard

Add rate limiter to all sources where bulk load is used

Open wrathagom opened this issue 7 years ago • 0 comments

ES has a queue limit for bulk operations so either I need to insert more rows/request or I need to put a rate limiter so that each request has a chance to finish before the next one starts. Ideally I'd add a loop of some sorts that only releases one more for each one finishing...

As it is right now inserts will fail if there are too many bulk requests.

wrathagom avatar Nov 13 '17 02:11 wrathagom