cqlite icon indicating copy to clipboard operation
cqlite copied to clipboard

Feature requests: Change Streams and File I/O Abstraction

Open Type1J opened this issue 3 years ago • 2 comments

Hello, I didn't see a better way to create a feature request, so I created this issue. I have 2 feature requests:

  1. Change Streams: Active queries, or queries that report new items (sorting can be disallowed on these queries) if the database adds, updates (old and new reported), or removes something that would have been included in a result set. This could also, just be a notification (callback or Stream entry) that the result set might have changed.
  2. File I/O should be a trait that is optionally passed in when opening the database. This allows storage in environments where storage is available, but the std library isn't implemented (embedded systems, wasm, etc.), and it could also be used to allow encryption to be added external to the database library.

Type1J avatar Dec 12 '21 23:12 Type1J

  1. Sounds interesting!

  2. I was previously thinking of having a networked pager, which shares transactions with a cluster of other deployments peer-to-peer (imagine running multiple servers and having the database sync between them automatically).

    I'm not sure I want to go down this road, but it might be a pretty useful feature. But if so, that might not work very well with alternate storage. Currently, this uses sanakirja as a pager and b-tree implementation. So supporting this would probably require either contributing this feature to sanakirja, or writing an alternative pager.

dyedgreen avatar Dec 14 '21 23:12 dyedgreen

For 2, let me be more specific in what I'd like to do.

Have you seen PouchDB? It allows one to store data using IndexedDB in a browser while providing a CouchDB-like interface. It also syncs data from a remote CouchDB just as CouchDB would replicate with another CouchDB.

I was hoping to provide a query on the server that publishes data to be synced with a subscriber on the client. If the mechanism is flexible enough to add to other databases, then I was going to use RedisGraph on the server side and cqlite on the client, but if your cluster idea is fast and reliable, then I may just use cqlite on both sides.

On the client, I'd like to be updated as data changes on the server (probably using either a gRPC-web stream, websocket, or server sent events connection), and when that change is added to the in-browser database, I wanted a notification, so I can redraw the UI.

I do something like this now with PouchDB, but I'd really like a graph database using Cypher on both sides. My main site logic is in Rust WASM, so cqlite would integrate well.

Type1J avatar Dec 15 '21 15:12 Type1J