Slow streaming from server to client with JSONstream

[noinkling]

I'm making an API request on the server and using JSONstream to parse and modify the (large) response, and then piping it to users using this module. The problem is that streaming to the user is extremely slow. JSONstream writes one object/array to its output stream at a time, and for some reason it takes around half a second for each of those to be sent over the socket - you can imagine how long the whole thing takes in my case where I'm sending thousands of (very small) arrays. If I simultaneously pipe to stdout it's incredibly fast as you would expect.

As a workaround I was looking for a way to get the "socket.io-stream stream" to buffer its input (or find a way to do it myself) so larger chunks can be sent, presumably speeding things up, but was unable to do so, short of concatenating the whole thing and sending it over at once, which basically defeats the whole purpose. If anyone has any ideas about how I could achieve this I would be grateful.

Changing between strings, buffers and object mode makes no difference.

Here's an example to help visualize what I'm doing (server-side):

// Response from API has structure like:
// { ...
//   dataset: [
//      ['2015-06-02', 4325],
//      ['2015-06-03', 3245],
//      ...etc.
//   ],
// ... }

const socketStream = ss.createStream();

const pipeline = request(someApiUrl)
  .pipe(JSONStream.parse('dataset.*', dataPoint => {
    // Map function
    dataPoint[0] = Date.parse(dataPoint[0]);
    return dataPoint;
  }).pipe(JSONStream.stringify());    // Can also omit this and use object mode

ss(someSocket).emit('new data', socketStream);
pipeline.pipe(socketStream);

If I just pass the API response straight through (no processing) it's much faster because it gets divided up using the default chunk size, rather than an array at a time.

[prabhu]

There is a magic configuration that will make every upload faster. Just set highWaterMark in bytes to 1.6Mb or something based on your network. This is documented under Stream api

ss.createBlobReadStream(f, {highWaterMark: 16384000})

[GoChartingAdmin]

@noinkling Did the above solution help? I am trying to implement a similar solution for my financial app

[noinkling]

@GoChartingAdmin To be honest all I can remember is that I gave up on getting it working, I can't remember if setting highWaterMark helped or not.

[GoChartingAdmin]

Thank you. Have you tried binaryJs library instead

Regards, Team DMarket

---

On Thu, 4/20/17, Malcolm [email protected] wrote:

Subject: Re: [nkzawa/socket.io-stream] Slow streaming from server to client with JSONstream (#84) To: "nkzawa/socket.io-stream" [email protected] Cc: "GoChartingAdmin" [email protected], "Mention" [email protected] Date: Thursday, April 20, 2017, 12:41 AM

@GoChartingAdmin To be honest all I can remember is that I gave up on getting it working, I can't remember if setting highWaterMark helped or not.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.