temporal icon indicating copy to clipboard operation
temporal copied to clipboard

Add scheduled membership changes

Open dnr opened this issue 2 years ago • 0 comments

What changed?

Add the ability for membership changes (joining and leaving rings) to be scheduled for a future time. For leaving, EvictSelf now accepts a time to make the eviction effective at. For joining, the join time is controlled by our ringpop wrapper factory and is harder to set, but a future PR will set it based on dynamic config.

Why?

This allows for more controlled membership transitions, and can reduce the effective delay between the first and last node seeing a membership change from 500-1000ms to 1-2ms. This is done by choosing a join time (or leave time) several seconds in the future, further out than the expected max propagation time. The join time is propagated to all nodes as a label, and they set a timer to re-compute their hash rings at that exact time (subject to clock skew and scheduling delays).

How did you test it?

unit test and lots of manual testing with local restarts

dnr avatar Feb 16 '24 08:02 dnr