Use SIMD where possible?

[thomasfoster96]

SIMD instructions are going to be available in ECMAScript 7, and are already supported in Firefox and Crosswalk. Plus, there's a Typed Arrays based polyfill available at https://github.com/johnmccutchan/ecmascript_simd. For projects using convnetjs, especially on mobile, SIMD might make everything a bit faster.

[karpathy]

interesting! thanks for pointing this out, I wasn't previously aware of it. I think the ultimate frontier of JS DeepLearning is still on GPU. It would be ideal if there was some Tensor object in JS with various operations that can happen on GPU or CPU. The CPU part could maybe use the SIMD instruction set for additional speed when on CPU. Cool stuff!

[thomasfoster96]

Perhaps Typed Objects (again, Firefox only for now) and Parallel JavaScript might be worth a look? Typed Objects could make going between CPU and GPU pretty easy.

I might fork the repo and see if I can get SIMD and Typed Objects working in Firefox.

[thomasfoster96]

@karpathy I've been playing around with Typed Objects and to me it seems to best way to use Typed Objects would be to have a VolType class that extends StructType. This would allow Vols to be very memory efficient and performant, and allow efficient arrays of Vols. Sounds pretty good to me :smile:.

This is a (broken) implementation of VolType:

function VolType(sx = 1, sy = 1, depth = 1){
  let VolType = new TypedObject.StructType({
    w : TypedObject.float64.array(sx).array(sy).array(depth),
    dw : TypedObject.float64.array(sx).array(sy).array(depth)
  });

  VolType.prototype.sx = sx;
  VolType.prototype.sy = sy;
  VolType.prototype.depth = depth;

  return VolType;
}

The VolType class could be used like this:

// For example, 32x32 RGBA images from a canvas
let ImageVol = new VolType(32, 32, 4);
let testImage = new ImageVol({w:imageData});

The above would be equivalent to calling:

let aVol = new Vol(32, 32, 4, randomImageData);

Typed Objects allows multidimensional typed arrays, so the add(), set() and get() methods wouldn't be needed, same goes for addGrad(), setGrad() and getGrad().

cloneAndZero() is also not needed, because another instance of a VolType is used instead. clone() is replaced as well, as passing a Vol to an instance of VolType would perform a clone.

The only huge downside to this is the Typed Objects are only available in Firefox beta builds, and as they are a part of the ECMAScript 7 standard, they won't be standardised until as late as September 2016. I'm sure Typed Objects could be polyfilled (just as SIMD and Typed Arrays can, just with lesser performance), but one would have to be written from scratch.

[karpathy]

Nice, thanks for looking into this.

Maybe it's possible to check if the user has TypedObject.float64 arrays available and if so, use them under the hood. We already do this with the implementation of zeros(), using typed arrays when possible.

Unfortunately, I think it's not very clean to put this in with the other code base because I traded speed for modularity, and have all kinds of math implemented inline, all over the place, and in hardcoded ways.

Maybe I should restart ConvNetJS from scratch... :)

[thomasfoster96]

I'd agree that using Typed Objects won't go very well with the current codebase.

I ended up actually sort of rewriting ConvNetJS anyway (I turned the concatenation of files into ES6 modules, and turned the pseudo-classes used into ES6 Classes), so you if you take a look at thomasfoster96/convnetjs the /src directory looks completely different. Not to mention using Typed Objects requires incompatible changes to the way Vols work.