sensu-go icon indicating copy to clipboard operation
sensu-go copied to clipboard

Background keepalive reconstruction

Open c-kruse opened this issue 2 years ago • 0 comments

Partial solution for https://github.com/sensu/sensu-enterprise-go/issues/2373

Closes: https://github.com/sensu/sensu-go/issues/4828

What is this change?

Instead of calling keepalived's initFromStore function to reconstruct failing keepalive switchset state in a blocking manner on backend startup, start it as a background goroutine and spread out the load across the default keepalive warning timeout of 120 seconds.

Why is this change necessary?

When many failing keepalives are present in an environment, this approach can put a heavy load on the etcd database immediately, and tends to flood the pipeline with keepalive events when they expire at the same time later.

Does your change need a Changelog entry?

Y

Do you need clarification on anything?

N

Were there any complications while making this change?

N

Have you reviewed and updated the documentation for this change? Is new documentation required?

N/A

How did you verify this change?

Performed a series of performance test comparing this strategy to the existing as well as others: https://docs.google.com/document/d/1s3xiZ4Dd_8UVZjUzpGGgjuH5aOfZu5dU1keq1ywXmS4/edit?usp=sharing

Is this change a patch?

N

c-kruse avatar Aug 03 '22 21:08 c-kruse