Make QuietDatabaseChecker able to check progress rather than just checking completion

[sfc-gh-bvr]

Prior to this change, there is a max-run-time specified for quiet-database checker. All conditions tested by quiet-database-checker have to be met within that timeout. When buggify is enabled, that timeout is sometimes not enough, and allowance was previously made by increasing the timeout when buggify is ON. However, a more tenable solution is to let quiet-database-checker continue as long as progress is being made, until some hard timeout limit is reached. This change implements a rudimentary version of that and applies it to DataDistributionQueueSize. There is scope for made the progress check more sophisticated and apply it to other conditions that are tested.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• [ ] The PR has a description, explaining both the problem and the solution.
• [ ] The description mentions which forms of testing were done and the testing seems reasonable.
• [ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• [ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 897b18f21d21c38db25605a4593ae96ba57180bb
• Result: :heavy_check_mark: SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

• Commit ID: 897b18f21d21c38db25605a4593ae96ba57180bb
• Duration 0:43:27
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 897b18f21d21c38db25605a4593ae96ba57180bb
• Duration 1:08:05
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 897b18f21d21c38db25605a4593ae96ba57180bb
• Duration 1:52:09
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)