[exporter/elasticsearch] Use batch sender exporter helper for reliability

[carsonip]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

This is a way to fix the issue reported in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/30792.

In case of a collector crash, the events buffered within the bulk indexer in memory will be lost, even with a persistent queue.

Describe the solution you'd like

Move the buffering from bulk indexer to exporterhelper batch sender introduced in https://github.com/open-telemetry/opentelemetry-collector/issues/8122, such that events are deleted from the queue only when they are flushed by bulk indexer.

Describe alternatives you've considered

No response

Additional context

I'm already working on a draft and related PRs are on their way.

exporterhelper batch sender https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is WIP and interface may change. Some features (e.g. batching by bytes) may not be available yet.

Blocked by https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32378

[github-actions[bot]]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

This enhancement makes sense to me, and sounds like a good idea. I'll defer to code owners in case there are component-specific thoughts here.

Removing needs triage.

[ycombinator]

This enhancement makes sense to me, and sounds like a good idea. I'll defer to code owners in case there are component-specific thoughts here.

Agreed, +1

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

Closing as completed via https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/34238 . Requires opt-in for the new behavior.