oboe.js

oboe.js copied to clipboard

I have a somewhat rare use case in that I'm making a static website (i.e. no webserver). All data is loaded via JSONP, since the browser will serve local files off the filesystem if they are javascript scripts, but will not serve HTTP requests to the local file system.

Thus I find myself having a 90 megabyte JSON string (loaded from compressed chunks off the file system). Parsing this with JSON.parse takes about half a minute and it locks up the page completely while it is running.

I'd like to use Oboe to parse the object in chunks, pausing 10ms after each chunk to give the browser room to breathe.

The documentation only specifies how to use Oboe with an HTTP request or a Node stream. However, I see this in the code:

function oboe(arg1) { 
    ...
    if( arg1 ) {
        ...
   } else {
      // wire up a no-AJAX, no-stream Oboe. Will have to have content 
      // fed in externally and using .emit.
      return wire();
   }
}

That sounds like exactly what I need. How can I go about feeding strings directly via .emit here? I haven't found any documentation to support this.

Ok, by poking in the source I've discovered it's just this:

var oboeStream = oboe();
// for every chunk of data:
oboeStream.emit('data', chunk);

I'd recommend a documentation improvement to show this!

Hi can you please go into how this can be used. I'm trying to use Oboe to read off a file input on the frontend without having to process on the server. So right now I have the file in the input, what is the next step?

Here is a gist of how it might be done using the FileReading API

https://gist.github.com/Aigeec/b202ae4866a9a6bd538dde57f5c30328

Update the path to oboe-browser.min.js to where ever your is.

I didn't really know about this feature, so definitely we could add some documentation.

That being said, it's strange for me that oboe has a few different apis: passing it a url and letting it make a request, passing it a stream and parsing the stream (node only), and this manual emitting. It might be worth considering how to simplify this long term.

One thing I've thought about is that oboe couples two concepts, initiating a stream and parsing a stream. Maybe the best solution would be a few related libraries that can connect together, individual ones for each version of stream initiation and another common one that does the parsing

I agree. I think the coupling make sense in terms of xhr where there is a lot of work to get it to 'stream'. Abstracting that away from the developer was a must.

It think with the availability of stream apis in both Node and the browser, I think you could just offer Oboe as a string transform stream and not bother with the request part in the core. Other libraries offer better solutions for that. We could still offer the xhr streaming part as an additional library for legacy or they can use <v2 for legacy browser support.

So basically the api would consume a json string stream and emit objects based on the requested node.

Arguably you could leave out the transformation and dropping aspects of the library and just let people do that in their own transformers.

As I was saying in the Promises PR I think you could simplify Oboe to two libraries. 1 for Node following the node stream api and a second for the browser following it stream api.

Its amazing how different the apis are.