Accessing BulkIndexerItem in OnError function

[youngstone89]

in esutil package,I would like to access BulkIndexerItem parameter in OnError function of BulkIndexerConfig struct instance so that I can produce dead letters when connection is gone. Will it be possible?

[Anaethelion]

Hi @youngstone89 sorry for the late reply.

There's indeed no way embedded to know which items were in the queue when an error is raised. You could however track the success of what you sent using the OnSuccess callback on the BulkIndexerItem and re-queue the ones that fails to acknowledge when you encounter an error.

I'll keep this issue open and take time to think this through. We could propagate back the items using the context but I would be worried about the memory consumption of this.

[rockdaboot]

You could however track the success of what you sent using the OnSuccess callback on the BulkIndexerItem and re-queue the ones that fails to acknowledge when you encounter an error.

This has the downside that you need to track all items that you Add() to the bulk indexer (which are millions in my case). If errors happen, this list of items can become very large quickly. A second point is the "When to re-insert" an item. It is not obvious. Should we wait 10s, 10m or 1h ? A good time might be after a flush - but that needs a AfterFlush callback. If we had such a function, the bulk indexer could just give it a list of failed items, which removes the need to track those items in the own code.

Update: just realized that there is a OnFlushEnd() callback in the BulkIndexerConfig.

[sumantha-kannantha-hpe]

Any updates on this?