connect icon indicating copy to clipboard operation
connect copied to clipboard

Elasticsearch output backoff should honor HTTP code `429`

Open p3r7 opened this issue 1 year ago • 0 comments

hi!

the elasticsearch output has an exponential backoff parameter, which is great!

sadly it only retries on HTTP codes 5xx (function shouldRetry).

Elasticsearch expects its client to also backoff with specific code TOO_MANY_REQUESTS (429) (doc).

this error code is somewhat nominal in the scenario of sudden message bursts and clients such as Logstash handles this gracefully (they refer to this as a "stalling output" scenario).

there could be 2 possible way to handle this:

  • add 429 as a condition in function shouldRetry
  • make the HTTP codes configurable, like the backoff_on parameter of the http processor

if that would help you, i'd be more than happy to submit a PR with the chosen fix.

finally, thanks for building this solution. it's very well thought off.

cheers

p3r7 avatar Jun 20 '24 13:06 p3r7