Roadmap discussion

[chrisui]

Firstly great work on Forerunner! Not actually used it yet but looked over all the docs, examples and code and looks solid. Recently been looking into a client-side query layer for data solution so this seems like it's going to be great for that.

I just wanted to kick off a discussion about the roadmap (as far as the readme states) to workout what's in progress and how features are going to look.

Before anything: "There are only two hard things in Computer Science: cache invalidation and naming things" - Phill Karlton

Data persistence on server-side

An idea I have here is to introduce an idea of 'persistance adapters' (or a better name). These adapters would describe methods for handling data changes in their own way. The default for the server side would be (as I've seen you mention elsewhere) an adapter which takes the actions on your ForemanDB (read, insert, update, delete etc.) and persist those actions into MongoDB. This would allow anyone to write (and share) their own adapter to work with any kind of persistance.

This would also work on the client-side where the default adapter could be a REST adapter.

Collection / query paging-- e.g. select next 10, select previous 10

I guess this is pretty trivial so I have nothing to add here.

Pull from server - allow client-side DB to auto-request server-side data especially useful when paging

This would fit in with the adapters described above however I see some complications with this as if it's auto-request across the board you lose the benefit of having the local database. What is interesting to explore here is storing the queries ran on the local database.

When a query is ran a second time we know we already have the result data in local database so don't need to wait on the request.

HOWEVER this is then using cached data which may have become invalid in the time between queries. How do we do this? It could just be always auto-requested by default if you implement an adapter and you can state which queries are cached. Leaving (for now) cache invalidation up to the user?

Really interested to see how you've planned to tackle this!

Push to clients - allow server-side to push changes to client-side data automatically and instantly

A persistance adapter concept would fit for this too and the default could be a socket.io implementation.

This feature would have to be quite flexible in order to aid scalability. For example I can't imagine very often you'd want to persist the entire database. Therefore it should be upto the client to 'subscribe' onto what changes they care about.

By default I imagine this would just subscribe to any changes to items in the the client-side collection storage. I think then it's upto the application specifics to declare other subscriptions.

Ie. I have a postCollection which is persisted between my client and server. I only request to get the top 5 posts ordered by number of comments. In this case I wouldn't want any automatic push to the client if I create a new post as it won't have invalidated my 'top 5 posts' query.

The cache benefit of Forerunner is the hardest thing with this server/client persistance.

Push to server - allow client-side DB changes to be pushed to the server automatically (obvious security / authentication requirements)

This would fit in with the persistance adapters described above and doesn't really have the issues (caching) you have when auto-requesting from a server. Any action on the client you probably want to make on the server side. It should of course be easy to opt out of this on a per action basis.

There'd need to be the appropiate events here to allow correct management of failures etc. I assume in some cases you'd actually want auto rollback? Say you deleted an item from a collection and for the sake of UX & responsiveness preemptively deleted it from local db - you'd probably want to revert that deletion upon a failure?

Replication - allow server-side DB to replicate to other server-side DB instances on the same or different physical servers

Haven't really got anything to add onto this myself.

So this is just a bit of a ramble of my thoughts which may be negated the work you're doing. My main point I think is Forerunner could be immense if we can take the main benefit (fast local db cache) and maintain that effectively with server-side persistance.

Just interested in seeing how you and others are imagining these features so we can push them forward and potentially I can get involved :)

Chris

Ps. On a side-note it might be nice to create a google group for discussions like these as well as promoting a tag on stackoverflow to use. I put this issue here so at least for now it would be public for anyone to contribute.