oboe.js
oboe.js copied to clipboard
jimhigson
Async Transforms
In the documents it mentions that you can transform the json streaming through in by returning a value from the callback. It would be nice if this operation could work async as well. We have a pattern in our micro systems where links are returned as a result and before passing this back to the user I like to follow the links server side so the user gets a set of hydrated results. Currently I have a rather ugly solution to this, but an async transform function would make this really simple.
@TSteele27 What's your current solution like? I know the current code waits for a listener to finish before the next gets started, so maybe what it could do is add a callback option to let that back-pressure mechanism know that it's getting an async transform.
I'm going to pick this issue as the one to put my thoughts into, though there are a couple related issues I've mentioned.
I think the code that would need to change is here, and there the related test are here.
The api I'm thinking about would be something like (note that arg2 and arg3 are already taken, but I'm not sure by what since couldn't find that in the API.
.node('pattern', function( matched, arg2, arg3, cb ) {
return asyncTransform( function(err, data){
cb(null, data);
}, 0);
}
})
But I'm having trouble figuring out how to fit that into the existing code. The existing code relies on having a return value, which I think would be undefined in the case of async transforms.
What if you returned an async function, promise or stream from the pattern listener? This would maintain the current pattern of returning something from the listener to indicate a transform and we would just have to add some code around handling the different async options.
Great ideas @TSteele27! That first one seems really promising to me. You could return a function that would be called by the library code, as opposed to actually invoking that function.
I really like the idea of using promises, although since Oboe is all ES5 and tries not to introduce dependencies, I'm not entirely sure if we could do that right now :confused:
Not entirely sure about how to do this with a stream, what would you have in mind?
Returning an async function is IMO the easiest of the 3 to implement. As for returning a promise I'm not convinced we would have to add a (production) dependency on a promise library since we would be using a promise returned by the client code, not generating a promise. As long as the promise library followed the promise spec would could rely on that without introducing a dependency. We may have to add a devDependency on a promise library to test to make sure that the code works, but I don't see a reason we would need to add a production dependency. Granted type checking a promise might be a bit difficult, but if we look for a .then method on the listener return value that would be a pretty good indication its a promise.
As for handling a stream, this one would be the most difficult, not quite sure how I would go about doing it since I am not intimately familiar with the oboe source code. The simplest thing to do would be to callbackify it in the library code and just handle it like async function, but if we wanted to support the actual streaming it would probably get more complex.
I agree strongly that promises should be an option if possible, even at the cost of introducing a dependency.
This 8k library provides "native promises only" as a polyfill. It's very bare-bones: https://github.com/getify/native-promise-only
I also need flow control in oboe as I'm reading a big json file form disk to stream it to the network.
@JuanCaicedo I think what you named arg2 and arg3 are documented here: http://oboejs.com/api#node-event, those are the path and ancestors of the matched parameter (named node in the doc).
Hey folks. Has there been any movement or solution found to this issue? I'm currently stuck on the same thing...
@lincolnthree nope, unfortunately I wouldn't count on their being any 😕There's not much work being done here
@JuanCaicedo Thanks. I ended up removing Oboe and chunking the file on the server side to get around this limitation. Can't actually stream properly if you're still loading the entire stream into memory. If there's no way to block the back-pressure then there's no point :/ Otherwise this library is great!
I'm interested in implementing this! @JuanCaicedo (or any other collaborators), do you have any pointers or thoughts about how to do it best?
Hi @wilg awesome, I'm excited that you're excited! I don't know exactly where to start, but @Aigeec might have more thought since he's looked at more of the files recently? If not, I can take a look and send you some thoughts 😀If you haven't gotten some feedback on this by Friday, feel free to ping me (I'm pretty forgetful 🙈😅)
Cool, trying to get my dev environment setup but am running into issues which I've documented in https://github.com/jimhigson/oboe.js/issues/174. Any help is appreciated. I'll keep digging around.
Instead of depending on promises, or hoping the user will return a promise-compatible object and trying to detect that, why not have another special helper value/method like oboe.drop?
oboe('/myapp/people')
.node('people.*', function(person) {
return oboe.await(function(resolve) {
setTimeout(resolve, 100);
});
});
Hi guys, any news on this? I too think oboe.await should be a great way of adding async callbacks without introducing dependencies on promises. In my case it would be useful to send data while streaming:
oboe('/myapp/people')
.node('people.*', function(person) {
return oboe.await(function(resolve) {
stream.write(person,(err) => resolve(oboe.drop));
});
});
Just adding my need for this as well. Something like oboe.await would be amazing. I'm streaming JSON results from a server (ultimately from Elasticsearch using ES's scroll method), and need a way to process nodes on the client asynchronously as they are received.
