add a streaming read/write API

[max-mapper]

napkin sketch:

var relayServer = require('relay')()
var database = require('database')
var ircConnection = require('irc-thingy')

database.pipe(relayServer.createReadStream())
ircConnection.pipe(database)

relayServer.createWriteStream().pipe(database)
database.pipe(ircConnection)

relay          <->     database <-> ircConnection

node server/webapp    indexeddb     websocket

[takempf]

Now that I'm looking into this issue I'm a bit confused. I read up on streams here (https://github.com/substack/stream-handbook) and I see how they could be useful for defining a well understood interface for a constant flow of data, but I'm not understanding how IndexedDB fits into this. AFAIK IndexedDB is client side, and I'm having trouble finding good implementations of the stream API client side.

Here's a quick run down of how the app works right now:

IRC Server (Freenode) <-> node-irc <-> socket.io <-> Backbone app

All I can really see streamifying there is the connection between node-irc and socket.io. My memory is a bit foggy now, but could you remind me what exactly we were trying to accomplish by adding this interface (IIRC it was to make it easier to add in modules which could interface with the app at large).

[max-mapper]

the goal is to separate the storage layer from the app layer so that people can write custom storage layers, so conceptually you just need to have the relay app js expose a read/write API that stuff can be plugged in to.

your default storage layer (what you have now) doesn't persist anything and just pipes socket.io into relay

another storage layer might be a polling XHR stream that pipes messages into relay

or you could have a websocket that pipes into indexeddb, and then indexeddb pipes into relay

the point is that if you just expose streams then you can layer the storage abstractions on top of the relay codebase. relay just sits there and waits for messages to get sent to it, and it sends messages back out when users do stuff in the app

i'm currently working on https://github.com/maxogden/level.js which will eventually work with the levelup API (work in progress)