opentelemetry-collector icon indicating copy to clipboard operation
opentelemetry-collector copied to clipboard

(exporterhelper): Add concurrency sender to send batches concurrently

Open moh-osman3 opened this issue 8 months ago • 1 comments

Description

This PR adds in a new component to the exporterhelper called the concurrency sender. This component is meant to limit the concurrency of RPC's independent from the queue sender's num_consumers. This functionality is desirable because the queue_sender does not apply backpressure, so the concurrency_sender can limit requests based on a configured concurrency limit. Additionally, the batch_sender can generate multiple batches from a request that are exported sequentially, so the concurrency_sender provides a way to export these batches in parallel.

This PR also changes the requestSender interface

type requestSender interface {
	component.Component
	send(context.Context, ...Request) error
	setNextSender(nextSender requestSender)
}

This allows us to optionally send multiple requests, which will allow batch producers to receive any errors associated with any of their exported batches.

Link to tracking issue

Related to https://github.com/open-telemetry/opentelemetry-collector/issues/10368

Testing

Added unit tests and performed local testing to ensure concurrency works and errors are propagated back to batch producers

moh-osman3 avatar Jun 28 '24 05:06 moh-osman3