copycat
copycat copied to clipboard
Optimize log iteration with per-replica pointers
Generally speaking, the Raft algorithm reads log entries sequentially. When replicating to a follower, the leader reads some number of sequential entries, sends a request, gets a response, and continues. The current log implementation is optimized for sequential reads for uncompacted segments. But for compacted segments, reading each entry is significantly more expensive due to binary search. This is somewhat mitigated by storing last read offsets and positions, but if the leader is reading from two different points in compacted areas of the log, this optimization will have much less significance.
Reading the log can be further optimized by storing the offset in the same way but on a per-replica basis. Essentially, we would just create something like a LogIterator of which one is created for each follower on the leader. This would allow sequential reads in almost all cases aside from the initial read.