replace Subscriber and MergeOperator with Transactors

[spacejam]

Subscribers suffer from internal queuing issues that are highly misuse-prone, and they only work for simple key prefix matching. Merge operators are a great idea but only work with a single key in a single tree at a time. Neither are transaction-compatible.

Goals:

• simple, familiar iterator combinator programming style
• arbitrary filter functionality, not just prefix matching
• misuse resistant - it should be hard to run out of memory or crash for some reason
• arbitrary source and destination trees for transforming data
• transactional atomicity for the entire sequence

more info to come as this is thought through more

[infogulch]

• arbitrary filter functionality, not just prefix matching

This makes me think of multipattern regex matching like in regex::RegexSet:

The key advantage of using a regex set is that it will report the matching regexes using a single pass through the text.

I'm imagining a scenario where each subscriber describes the keys they want to listen for changes on as a regex, and they're all compiled together into a single regexset which can then be efficiently queried to find all subscribers that match each changed key. Though perhaps leaping from "simple prefix match" to "full regex engine" is a bit much. 😄

• transactional atomicity for the entire sequence

I could imagine wanting to subscribe to key changes that occur at different times during the transaction lifecycle for different reasons:

1. During the transaction, say, to implement a custom index that is never out of date. Kinda like a SQL index or trigger. Here I'd want to be able to add new events to the same transaction before commit. I think your statement above would provide for this.
2. After the transaction commits, say, to update a remote client that is listening on an open network socket that their data has changed. Here I may not want to stall the transaction waiting for a response over the network, but it would still be nice to be able to notify them instantly without having to poll for changes. Could also be used to create a read-replica, etc.

[ckaran]

One thought; you were talking about updating sled to 1.0 sometime soon (like, this week) at one point, would this be a breaking change to the API? That is, is it worth putting off 1.0 for this?

[ckaran]

arbitrary source and destination trees for transforming data

Does that mean arbitrary sets of trees? That is, more than just one source and one destination? Given that transactions currently support arbitrary sets, I think the answer is 'yes', but I just wanted to confirm that it's true.