Add support for time-based snapshot intervals

[pcholakov]

This change adds a basic time interval configuration option for triggering automated snapshots. When both time and record-based intervals are set, the logic for the trigger condition is an "and" between the two, so the record interval becomes a minimum requirement for time-based snapshotting.

Time interval-based snapshotting will get a lot more useful with incremental snapshot support, but this is already a big improvement as it allows operators to configure snapshots in slightly more intuitive terms of elapsed time.

The time interval is measured from the created-at timestamp of the published snapshot, which is based on the wall clock time of the producing node. Clock drift error is assumed to be relatively negligible compared to typical snapshotting intervals. A small amount of jitter is added to each partition's snapshot age check to spread snapshot uploads out over time.

Configuration examples

Periodic + minimum number of records

[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval-num-records = 10000
snapshot-interval = "30 min"

Translates into: "snapshot every 30 minutes, as long as the applied LSN has advanced by least 10,000 records since the last snapshot".

Periodic (unconditional)

[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval = "24 hours"

Translates into: "snapshot once every 24 hours even if zero new changes are committed".

[github-actions[bot]]

Test Results

  7 files  ±0    7 suites  ±0   2m 37s ⏱️ +7s  47 tests ±0   47 ✅ ±0  0 💤 ±0  0 ❌ ±0  200 runs  ±0  200 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit ba47e395. ± Comparison against base commit dbb33cf5.

:recycle: This comment has been updated with latest results.

[pcholakov]

Thanks for taking a look, @tillrohrmann! Good points, addressed both. Docs will definitely need an update but there are a couple more PRs in this train and I'll take a pass all the changes are in.