sled icon indicating copy to clipboard operation
sled copied to clipboard

API for checkpoint / snapshot / backup system

Open thedodd opened this issue 4 years ago • 1 comments

Use Case:

Rocks equivalent:

  • https://github.com/facebook/rocksdb/wiki/Checkpoints
  • https://docs.rs/rocksdb/0.15.0/rocksdb/checkpoint/struct.Checkpoint.html

The primary use case here is exactly as described in the Rocks use case. Full & incremental backups. In my specific use case, I am building a distributed data storage system which uses async-raft for consensus, which includes a protocol where snapshots are sent to new nodes to bring them up-to-speed. Good stuff.

Proposed Change:

The API I am envisioning is as follows.

  • sled::DB gets a new method checkpoint(path: impl AsRef<Path>), which does the work of generating a new DB checkpoint which will be written to the given path.
  • Given the nature of checkpoints, it seems logical that this method only be exposed on the DB type and not on Trees (maintainers would definitely know better on this one).
  • Some discussion is merited around incremental checkpoints / backups and what patterns would be best for this.

Who Benefits From The Change(s)?

Anyone and everyone looking for full / incremental backups, and the various use cases which emerge from that capability.

Alternative Approaches

Instead of calling this new API method checkpoint, we could call it backup.

thedodd avatar Nov 25 '20 04:11 thedodd

@thedodd Take a look at https://github.com/spacejam/sled/issues/587#issuecomment-580276392 as well as the rest of that issue. Can we combine all of the ideas into a single new feature request?

ckaran avatar Nov 25 '20 15:11 ckaran