Add scheduled membership changes
What changed?
Add the ability for membership changes (joining and leaving rings) to be scheduled for a future time. For leaving, EvictSelf now accepts a time to make the eviction effective at. For joining, the join time is controlled by our ringpop wrapper factory and is harder to set, but a future PR will set it based on dynamic config.
Why?
This allows for more controlled membership transitions, and can reduce the effective delay between the first and last node seeing a membership change from 500-1000ms to 1-2ms. This is done by choosing a join time (or leave time) several seconds in the future, further out than the expected max propagation time. The join time is propagated to all nodes as a label, and they set a timer to re-compute their hash rings at that exact time (subject to clock skew and scheduling delays).
How did you test it?
unit test and lots of manual testing with local restarts