ml5-next-gen icon indicating copy to clipboard operation
ml5-next-gen copied to clipboard

Neural network support webgpu

Open ziyuan-linn opened this issue 1 year ago • 4 comments

This PR adds webgpu support for neuralNetwork so it can be used without called ml5.setBackend.

Changes:

  • bump TensorFlow.js dependencies to a version that supports readSync as @lindapaiste suggested.
  • add await tf.ready() to the init function of DiyNeuralNetwork @lindapaiste suggested.
  • change copy and crossover to async, support both promise and callback interface.
  • Update example sketches accordingly

The ml5.neuralNetwork function is technically an async operation like with other models. It should be called with a callback function or placed within p5 preload function. Our examples, however, have been treating ml5.neuralNetwork like a synchronous function since it sometimes (when not loading data) behaves synchronously. The changes from this PR adds await tf.ready() thus making it definitively asynchronous.

The functions copy and crossover also instantiate the neuralNetwork object, which is why they have to be async.

I updated all example sketches, removing ml5.setBackend and moving the ml5.nerualNetwork function to preload for non neuroevolution examples.

The neuroevolution examples are kept on cpu backend for performance. Adapting for async copy and crossover slightly complicates those examples but I think they are still fairly readable. The ml5.neuralNetwork calls in neuroevolution examples are still synchronous, but they seem to not cause any error for now. I will try to come up with a clean way to adapt them to async.

ziyuan-linn avatar Jun 15 '24 01:06 ziyuan-linn

Hi @ziyuan-linn, maybe we can meet to discuss these! I would very much prefer to keep the synchronous calls for neural network especially for the neuroevolution examples. I originally made asychrnous versions of these examples for teaching the class and the students really struggled with them along managing the draw() loop and other aspects of working with neural networks inside animated systems. The examples are also going to be printed with the synchronous code in the nature of code book in chapters 10 and 11 (along with an explanation.)

I realize it's not typical to use synchronous calls for ML models, but since these are tiny neural networks running on CPU for these p5.js examples I think it's worth making an exception and hacking our way around it!

Later, when p5.js adopts promises I think we could reconsider!

shiffman avatar Jun 17 '24 20:06 shiffman

I stumbled across this, as @ziyuan-linn mentioned, that explanation was nice and clear, @shiffman, thanks!

Out of curiosity, is the adoption of promises in p5 in the works / planned?

jchwenger avatar Sep 21 '24 13:09 jchwenger

Out of curiosity, is the adoption of promises in p5 in the works / planned?

Looks like it is! https://github.com/processing/p5.js/blob/dev-2.0/rfc_p5js_2.md#async

ziyuan-linn avatar Sep 21 '24 18:09 ziyuan-linn

Awesome, thanks @ziyuan-linn !

jchwenger avatar Sep 22 '24 15:09 jchwenger