spacejam
access to log for replication
Since the underlying structure appears to be a log, i was wondering why there is no api to access that log. Is it missing or am i confused about the purpose of sled?
I was expecting sled to be easy to use in a "backup" type of replication setup, with only occasional syncs, for embedded devices.
unfortunately there's nothing here that looks like you can do that https://docs.rs/sled/
Hey! This is actually a big priority for me, because once we have a nice log tailer API, we can trivially implement replication. One hurdle to get there is the way that huge values are written. Currently, we are simply throwing them in their own file, off-log. We do this because the log is split into fixed-sized segments so that we can track garbage collection information on a per-segment basis, and then identify segments that we can reuse for future writes. But we can change the underlying log API to support allocation of several segments for a single write, which would facilitate this nicely, and then allow these huge values to also be transmitted to any log tailers in the same way as everything else. The final log-related change that I'd like to see before 1.0 (when I'm making a forward-compatible binary format guarantee) is group commit, where several log writes are able to specify that they are part of the same batch, and if some are missing, the whole batch is torn, and recovery stops at this point.
If you're interested in more directly accessing the log, you may be interested in the pagecache crate that the sled tree is based on top of. This provides low-level access to the underlying log, and also lets you write your own lock-free structures on top of the persistence layer. The sled tree is actually not very much code on top of the pagecache.
thanks for the long response!
pagecache looks a little too low level for me. I did some experiments with it, but didnt understand how the log and page fragments actually fit together. Log appears to be a high level implementation over pagecache that creates an append-only log, rather than being a way to replicate insert/replace/delete operations.
maybe its because the docs dont render properly, but i'm just super confused how anything fits together.
@aep the first real first-class access to mutations is landing in #460, although this is more of a logical stream of updates to KV pairs than the future goal of a binary replication stream as well. But maybe it's helpful for your use case!