require('bpipe').std{in, out}

[dominictarr]

Okay so I have this module that takes a csv stream and outputs a graph. https://github.com/dominictarr/line-graph

I would love it pipe to it with bpipe, but i'd need a way to push data into that particular instance but handle it in a custom way, instead of bpipe's default selector pattern. I can think of some other things you could create like this too, like what about browser push deploy? I could pipe a html file into a "hypervisor" window, which would then "spin up" an iframe and run that code in... perhaps the response stream would be the console.log output or something.

[Marak]

@dominictarr -

I've been working on decoupling bpipes internals and trying to figure out the ideal API implementation. For reference the v0.2.0 branch has much cleaner code.

For your first use case ( piping the csv ), I am trying to understand where you want to the pipe the results. line-graph will output a image? In which custom way will this need to be handled? Is piping the result to an image tag not sufficient?

Can you provide some pseudo-code of what the developer API might look like for the CSV example?

As for browser push deploy, could you also provide a pseudo-code example of that as well? bpipe now targets the body tag by default if no selector is provided. You speak of hypervisor and iframe, is this to protect scope of page incase previous code pushes need to be cleared out for next push?

It's important we iron out the potential use-cases before pushing forward too far on development. I suspect some functionality will be split into separate modules while other APIs will need to be improved so other modules can consume them.

[dominictarr]

okay so I want to generate the image in the browser - currently line-graph runs in the browser, or with node-canvas. node-canvas is good, but it's a bitch to get installed... I'd like to pipe data into an instance and get a visualization - of course there are many other cases where this would be cool.

okay so I guess since, really, it would be possible for multiple connections to come into one browser instance it might look like this:

var LineGraph = require('line-graph')
var canvas = document.getElementById('mycanvas')
var createServer = require('bpipe'')
createServer(function (stream) {
  stream.pipe(LineGraph(canvas))
}).listen('ws://bpipe.io/#hbrjobertorektmoe')

here the url is the address of the bpipe server + a uuid that represents this specific instance. I suggest a server pattern here, because a server is a thing that receives zero or more streams, at undefined points in the future. Of course, it's not really a server, it's a server tunneling through a client, but the important thing is that it feels like a server. Another place this pattern works is https://github.com/dominictarr/reconnect reconnect attempts to reestablish a connection when it drops. This differs from a normal server because the connections are never in parallel - but that is okay, because a serial sequence of streams is a subset of parallel connections - it's just a server that happens to only ever have 1 connection.

the push deploy "hypervisor" could be very similar...

require('bpipe')(function (stream) {
  stream.pipe(concat(function (source) {
    //attach a preamble that adds some constants, such as
    //the bpipe id we want that instance to have...
    var childId = generateId()
    var url = 'dataurl:'+ toBase64(preamble(childId) + source) //etc etc
    document.body.append(createIFrame(url))

   //or open a window, or create a web worker, or whatever
   //now, send the child's id back to the pusher, so they can connect to it.
   stream.write(childId)
   stream.end()
  })

}).listen(...)

something approximately like that?

[dominictarr]

oh just reading the code... I notice you can only have one stream into each browser instance, what about using mux-demux (or https://www.npmjs.org/package/dataplex) then it would be possible to have multiple streams over one websocket stream.

[Marak]

@dominictarr - Examples look good, getting the API closer to that should be easy.

As for duplex, bpipe is currently using one stream and delegating based on the contents of the JSON. Is that the poor mans multiplexing? I think using mux-demux would make a good addition. I've looked into dataplex before. It looks promising, but I don't want to depend on it right now.

Will mux-demux be sufficient for pushing around data? If so, I'd rather keep it simple.

[dominictarr]

Oh right. per message multiplexing is okay really, and that is basically what mux-demux (and the others) are using internally. mux-demux will work. it is something that I have been meaning to rewrite for ages, but havn't gotten around to yet. Basically the problem is that none of these options handle back pressure per stream... which seems like a thing you'd want.

[Marak]

@dominictarr - I went ahead and implemented dataplex as stream router. It's working pretty well so far.

I think basically what is needed is pattern for easily adding new pipes / transform streams to client / server / browser. Everyone will want to make custom transform logic, best to make that as easy to possible to do. I implemented basic system for adding custom stream logic to bpipe.

The current server API is looking something like this:

server.start({ port: 8001 }, function (err, server){
  server.on('wsconnection', function (stream){
    stream.plex.add('/querySelector', require('../lib/pipes/querySelector')());
  });
  console.log('bpipe ws server started on port ' + server.address().port);
});

This will setup a bpipe-server and add a new multiplex channel called querySelector. querySelector is now (somewhat) modularized as ./lib/pipes/querySelector. Inside this folder contains logic for handling: client, server, and browser logic for querySelector binding.

The current client API now routes based on the incoming ws path.

bpipe localhost:8001/querySelector

This will open up the querySelector pipe on the querySelector multiplex channel. If no path if specified, querySelector is the default.

The browser API would then look something like:

var bpipe = require('bpipe');
bpipe.connect();
bpipe.open('/querySelector').pipe(doStuff);

---

I'm still trying to sort out which functionality will be common to all ./lib/pipes and what is specific to each pipe. Adding the graph functionality would be a matter of creating a custom multipex channel with specific logic for handling graph.

Can you look over current the architecture and see how this can be further refactored for ease of use?