A Spark is not a Readable Stream

[aslakhellesoy]

It has no read([size]) method.

If it had, I'd be able to easily hook it up to ShareJS like this:

var share = sharejs.server.createClient({backend: backend});
primus.on('connection', function (spark) {
  share.listen(spark);
});

That doesn't work - I get this exception:

ShareJS/lib/server/session.js:332
  var req = this.stream.read();
                        ^
TypeError: Object #<Sparky> has no method 'read'
    at Session.pump (ShareJS/lib/server/session.js:332:25)

[3rd-Eden]

As we already discussed over IRC, this is because Primus implements the Stream1 interface instead of Streams2. The primary reason for this was for 0.8 support as well as performance benefits. But it's certainly something that should be done once node 0.11 is released.

[aslakhellesoy]

Ok. Would it be fair to provide a spark.stream() function to get a Streams2 compatible Duplex wrapping the Stream1 compatible Spark object? In your already stellar JSDoc you could label this function as "Requires Node 0.10 or higher".

[davedoesdev]

With a Streams2 interface, would we get help with back-pressure?

[3rd-Eden]

@davedoesdev the back-pressure should be handled by transport/transformer that you're using.

[davedoesdev]

That makes sense but I got confused with this in the README:

This method always returns true so back pressure isn't handled.

[davedoesdev]

@3rd-Eden Do you know which transports handle back-pressure? IMHO this is a pretty important thing to support. From the code, it looks like ws doesn't, for example.

EDIT: This seems relevant: https://github.com/maxogden/websocket-stream/issues/1

[davedoesdev]

So I'm thinking about back-pressure again. How does the transformer help here without exposing it to the app in the API? I wonder if the whole emitter way of exposing messages is wrong?

[davedoesdev]

I'm coming to the conclusion that until https://github.com/whatwg/streams gets into browsers, it might be best to implement back-pressure over the top of streams1/browser streams. Probably this can be done in a module independent of Primus but which can be used with Primus. May not be very efficient though.

[davedoesdev]

@aslakhellesoy @3rd-Eden this is what I came up with: https://github.com/davedoesdev/primus-backpressure

[3rd-Eden]

I'm just wondering if it's worth all the extra overhead which streams2 adds (which is quite a lot imho) or that this serves better as a plugin as David has done.

On Friday, August 1, 2014 at 10:44 AM, David Halls wrote:

@aslakhellesoy (https://github.com/aslakhellesoy) @3rd-Eden (https://github.com/3rd-Eden) this is what I came up with: https://github.com/davedoesdev/primus-backpressure

— Reply to this email directly or view it on GitHub (https://github.com/primus/primus/issues/121#issuecomment-50862279).

[davedoesdev]

Yes - primus-backpressure adds overhead because it has to send status messages letting the sender know how much data is free in the recipient's buffer. There are obviously plenty of use cases where this isn't necessary because primus is being used in anger without it!

Without a proper streams2-ish API from all transports, Primus would always have to do backpressure over the top of them anyway.

Eventually https://github.com/whatwg/streams may give us something in the browser.

[davedoesdev]

Just a quick pointer. I used a similar technique to do multiplexing (with back-pressure for each stream separately) over any streams2 Duplex:

https://github.com/davedoesdev/bpmux

It works over Primus (via primus-backpressure).

[davedoesdev]

Not sure on the progress of whatwg streams. This issue has been open a while now. Anything different now?

[lpinca]

AFAIK the streaming interface has been left untouched.