Enable a mode where a txn-commits wait for on-disk-persistence

[shriraml-ms]

Is your feature request related to a problem? Please describe. Yes, currently a txn can be globally committed without having the writes hit the disk because the commit doesn't wait for writes to be flushed to disk. This means that during DR, a previously globally committed txn can be lost. In systems that require higher guarantees on durability, it would greatly help if the ccf-commit flushed and waited for the writes to hit the disk. Additional comments:

• This feature would obviously come with some perf-penalty. It would be good to have a perf test that shows comparative perf & have the mode/flag come with a pointer to documentation that lists the approx perf penalty. This would allow users to make the right tradeoff for their app.
• In cases where the host storage is malicious or misbehaving, the benefits to durability from this feature are obviously questionable - this is something that needs to be documented as well.
• I'm not sure if introducing an untrusted entity (host/storage-driver) into the critical path of commit, introduces any other attack vectors - this likely needs to be understood as well.

Describe the solution you'd like Add a configuration switch that tells CCF to flush and wait for writes to hit the disk before marking a txn as globally committed.

Describe alternatives you've considered One alternative is to run CCF across many more nodes but this can get expensive quickly.

Additional context n/a