[ZOOKEEPER-3657] Implementing snapshot schedule to avoid high latency issue due to disk contention

[lvfangmin]

[asf-ci]

Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build-maven/1761/

Failed Tests: 1

PreCommit-ZOOKEEPER-github-pr-build-maven/org.apache.zookeeper:zookeeper: 1

• org.apache.zookeeper.test.RestoreCommittedLogTest.testRestoreCommittedLogWithSnapSize

[lvfangmin]

retest maven build

[hanm]

interesting. two quick questions:

• zookeeper has some randomness in today's sync processor when snapshotting already, i am curious to know what's the short comings of existing approach that motivates this change.

• i don't see observers processing snap ping packet in this pull request so only quorum servers will be scheduled by leader for snap shot generation?

[anmolnar]

@lvfangmin I second @hanm 's questions. This patch is quite huge, are you still working on it?

[lvfangmin]

@hanm the randomness snapshot could introduce high latency if majority taking snapshot at the same time, when the total DataTree size increasing, it will take longer time to do snapshot, which means it's more likely majority will take snapshot at the same time with longer period. Which will be a problem when running ZK on a single disk driver.

From what we saw in benchmark, the write throughput within SLA for 6GB DataTree size is more than 10X smaller than 100MB DataTree.

That's why we introduced this feature. Observer don't need to handle SNAPPING, since the quorum ack latency is only affected by participants.

@anmolnar this feature is complete, it has been on our prod for more than 6 months.

[lvfangmin]

Rebased and addressed the nit suggestion from @anmolnar.