oboe.js
oboe.js copied to clipboard
jimhigson
Pause and resume?
Hello. I have a situation in which I need to coordinate two sorted streams which means that I must pause and resume to align them.
Is it possible to integrate a pause and resume function? I tried to pause the source readable stream that was feeding into oboe, but oboe's parser seems to be fast enough that I cannot get oboe to consume only one item at a time.
Thank you.
Hmm, I think the node function has a back pressure mechanism. That is, it'll wait for a listener to be completely finished before it moves on to the next one. Maybe you could just use setTimeout inside the listener to slow down the parsing?
Hi, Juan. I looked into it but I didn't think it would work for my situation. I need to reliably guarantee that I would be able to "catch" and individually handle oboe's processing of every row of data because I need to match it up with another stream. A timeout might work on one system but would fail somewhere else.
What I ended up doing is writing each row from oboe into a new pausable stream and then working with that.
There are a few issues were we've discussed adding a callback to the node function to enable async transforms (#81, #88). Do you think this would address your use case? If so, we could batch them all together.
@JuanCaicedo Would help me too. Being able to use async methods inside oboe events would be awesome. Is somebody working on this or is it at least on the roadmap? If not I might start a branch in the near future.
I am fairly new to nodejs but not making async available inside oboe events is kind of saying we do not really support NodeJs. Isn't nodejs based around the whole idea to have async automatically by using callbacks/promises so the event loop can schedule nicely. Also if you want to use methods/libraries that do not offer synced method, as many do not, you either cannot use it or you are forced to implement dirty workarounds. That latter was actually my case.
Don't get me wrong Oboe.Js is really awesome and I am able to write very small/clean code to transform even very big jsons.
