apache
[NEW] Allow forcing use the full sync when the lag was too large
Background
Currently, we're using the WAL to do the partial sync like the Redis backlog, as well as the write buffer between worker threads and feed slave thread. Users should tune the WAL size with the key-value size and it's hard in most cases, since the user key-value size may be varied over time. We prefer to give it a generic larger value to the WAL to fit into most cases, but every coin has two sides, it may slow down the partial sync when the key-value size was small and failed to trigger the full sync.
Proposal
The core problem was the generic WAL size may cause the partial sync too slowly when the key-value size was small, and it's hard to adjust the WAL size instance by instance since the size may be changed over time. So we need to find another way to work around this case, and the lag threshold should be a good choice since increment logs were synced by sequences.
TODO
- [ ] Add the
force-full-sync-lag-thresholdconfiguration to specify and enable the full sync lag threshold, default is 0 which means disabled - [ ] Check the next seq, if it's a new slave and overrun the full sync lag threshold, let's use the full sync mechanism instead of the partial sync
- [ ] Add test cases
cool idea from @Alfejik
I don't think it's good idea to add this config ,in fact , what we need is repl-timeout,when timer exceed repl-timeout, close the conncetion and do nothing.fullsync may cause io issues
@smartlee I have added more detailed information and high IO was expected in this scene. repl-timeout should be another issue, can u help to describe the context about when and why we need it.