data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Improve OpenSearch bulk retry logging

Open travisbenedict opened this issue 11 months ago • 1 comments

Is your feature request related to a problem? Please describe.

It's difficult to track bulk retry failures if Data Prepper is writting to multiple indexes.

Describe the solution you'd like

  1. Log the bulk failure the first time that there is a failure instead of only doing it every 5 times.

  2. Update logging to show the OpenSearch index for which the bulk request failed.

https://github.com/opensearch-project/data-prepper/blob/ed3f75ed3beedfb6d72af1f4b63ae0d085f72a0b/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/BulkRetryStrategy.java#L243-L244

Describe alternatives you've considered (Optional) N/A

Additional context N/A

travisbenedict avatar Mar 12 '24 16:03 travisbenedict