Asynchronous Packing and Unpacking

[quintopia]

Given that a major use case of such a tool is to decrease the time spent downloading a large file, instead spending that time decoding that file, it would seem appropriate that the unpack function be asynchronous. e.g. that it interrupts itself occasionally to let the browser run (using a settimeout callback, perhaps one that calls back to itself right at the beginning of each recursion of recursiveUnpackerParser) and allows its progress to be externally tracked (perhaps by allowing the user to set an onprogress event listener). Perhaps this would not necessarily always need to be the default behavior, but it should be an option.

[rgcl]

You can convert any function to asynchronously simply with setTimeout, example:

function jsonpackPackAsync(json, next) {
  setTimeout(function() {
    var packed;
    try {
      packed = jsonpack.pack(json)
    } catch(e) {
       next(e);
    }
    if(Boolean(packed)) {
      next(null, packed)
    }
  }, 0)
}

// Example of use:
jsonpackPackAsync(json, function(err, packed) {
  console.log(packed);
})

In any case it is a good idea! I think the ideal would add the methods jsonpack.packAsync(json, callback) and jsonpack.unpackAsync(packed, callback)

What do you think?

(Consider setImmediate )

[quintopia]

Maybe I am not very knowledgeable about javascript, but I'm not sure how you can make jsonpack.pack return before it is finished packing without actually modifying jsonpack.pack. To me it looks like this jsonpackPackAsync just runs jsonpack.pack after the I/O queue is clear, and then doesn't give up cpu control until the json is completely packed. Which means all events that get queued after jsonpack.pack gets called will have to wait on it, freezing the page. Am I incorrect?

On Thu, Aug 6, 2015 at 11:41 PM, Rodrigo Gonzalez [email protected] wrote:

You can convert any function to asynchronously simply with setTimeout, example:

function jsonpackPackAsync(json, next) { setTimeout(function() { var packed; try { packed = jsonpack.pack(json) } catch(e) { next(e); } if(Boolean(packed)) { next(null, packed) } }, 0) } // Example of use: jsonpackPackAsync(json, function(err, packed) { console.log(packed); })

In any case it is a good idea! I think the ideal would add the methods jsonpack.packAsync(json, callback) and jsonpack.unpackAsync(packed, callback)

What do you think?

(Consider setImmediate http://stackoverflow.com/a/24119936/1044759 )

— Reply to this email directly or view it on GitHub https://github.com/sapienlab/jsonpack/issues/14#issuecomment-128581537.

Regards, David Rutter

[rgcl]

Good question, depend of the virtual machine where javascript run, but all have in common that uses a continuous event loop in a single thread, that loop simulates multithread in a single thread, so the important topic here is not to block that only one thread.

I think that with setImmediate or (if setImmediate is not implemented) setInterval we already not block the thread (only block the "soft" thread).

Of course, we can use a grid but it is out of the scope of jsonpack at the moment (that can be supplied "ad hoc" like anything else :)

The native implementation in C is an interesting TODO (If anyone is encouraged would be great!)

After thinking i think that jsonpack can be sync and async:

• jsonpack.pack(json) <- synchronously
• jsonpack.pack(json, callback) <- asynchronously

In that case, this two examples will be valid:

// sync:
var packed = jsonpack.pack({ name: 'pepe' });
var json = jsonpack.unpack(packed);

// async
jsonpack.pack({ name: 'pepe' }, function(err, packed) {
  jsonpack.unpack(packed, function(err, json) {
    // json
  });
})

If the last parameter is a function, then work in async mode, else in sync mode.

[rgcl]

Anyway I'm also not sure 100% about this, before approving this I need to do some tests with large files to see if the asynchronously blocks the rest of the I / O events (as you say). Just in case ...

[quintopia]

I like the idea of an overloaded pack and unpack. (moreso the unpack, of course). I'm not clear on the purpose of the callback parameter though. What should the function you pass be for? What does it do? On Aug 16, 2015 2:48 AM, "Rodrigo Gonzalez" [email protected] wrote:

Good question, depend of the virtual machine where javascript run, but all have in common that uses a continuous event loop in a single thread, that loop simulates multithread in a single thread, so the important topic here is not to block that only one thread.

I think that with setImmediate or (if setImmediate is not implemented) setInterval we already not block the thread (only block the "soft" thread).

Of course, we can use a grid but it is out of the scope of jsonpack at the moment (that can be supplied "ad hoc" like anything else :)

The native implementation in C is an interesting TODO (If anyone is encouraged would be great!)

After thinking i think that jsonpack can be sync and async:

• jsonpack.pack(json) <- synchronously
• jsonpack.pack(json, callback) <- asynchronously

In that case, this two examples will be valid:

// sync:var packed = jsonpack.pack({ name: 'pepe' });var json = jsonpack.unpack(packed); // async jsonpack.pack({ name: 'pepe' }, function(err, packed) { jsonpack.unpack(packed, function(err, json) { // json }); })

If the last parameter is a function, then work in async mode, else in sync mode.

— Reply to this email directly or view it on GitHub https://github.com/sapienlab/jsonpack/issues/14#issuecomment-131495440.

[bchr02]

+1 on true async (non-blocking) support

[deviousasti]

Just run in it a webworker?