Predictions in Batch

[jbigatti]

Hi!.

How could I do predictions in batch?

Taking mnist-cnn as example, how could I do model.predict([Float32Array[784], Float32Array[784], Float32Array[784]])?

Just for one input is working: model.predict({input: [Float32Array[784]]})`

Thanks a lot!

[transcranial]

Yep, right now, predict() takes in single inputs, so to predict on a batch, you can do something like (model.predict returns a Promise):

// using bluebird Promises
let batch = [a, b, c] // where a, b, c are all Float32Array(784) inputs, for example
Promise.mapSeries(batch, arr => model.predict({ input: arr }))
  .then(ouputs => {
    // ...
  })

or in an async function:

// using async/await
for (let arr of batch) {
  let output = await model.predict({ input: arr })
}

[jbigatti]

Great!. Thanks!

[unsalted]

I don't think this would work or apply to a sequential batch input if I'm not mistaken. Is there a solution for predict_on_batch in that instance?

[transcranial]

@unsalted you mean in a Sequential model? The input data object is keyed by the names of the input layers, or just input for Sequential models, so the above example actually represents what running batches in a Sequential model would look like.

[unsalted]

Where the data is sequential time based data that is trained in batches where sequencing and "statefulness" matters (think large char-rnn or word-rnn).

[unsalted]

I'm relatively new to keras so forgive me if I have something confused.

Btw y'all have been on a roll squashing bugs today!

[koampapapa]

Edit; i was able to get something working using a for loop and a callback like in http://stackoverflow.com/questions/10004112/how-can-i-wait-for-set-of-asynchronous-callback-functions .

I followed @transcranial's example for predicting in batch. But the model makes the same output step for each unique input step. This output is not just the same within batches, but across batches too. The output will be the same number if I put the following script on repeat forever. When I switched to single inputs (one-by-one), it did start producing random outputs.

function randomArray(callback){
	var output=[];
	for(var x =0; x < 13; x++){
		output.push([]);
		for(var y =0; y < 26; y++){
			for(var z=0;z<39;z++){
				output[x].push(Math.random());
			}
		}
		output[x]=new Float32Array(output[x]);
	}
	callback(output);
}

randomArray(function(batchInputArray){
          Promise.all(batchInputArray.map(arr => model.predict({ input: arr })))
		  .then(outputs => {
				var cP = [];
				for(var a =0; a < 13; a++){
					cP.push(outputs[a]['output']);
				}
		    		console.log("Here are batch currPredicts");
				console.log(cP); // will be 13 identical floats
                             
		  })
 })

Any suggestions? I have my batch input size set to (13,26,39) in the initializer for python. The output is just a single floating point number. (batchSize,1). When you use a batchSize of 13, the output becomes (13,1).

My browser doesn't seem to have the await function to try that example?

[transcranial]

Edited my comment above as it was misleading. Promise.all runs each item in parallel, and that does not work unless you have multiple instances of the model available and you do some sort of concurrency management. The async/await option does work though.

Alternatively, you can use Promise.mapSeries with the bluebird Promise api. I just tested this and it works. Replace your Promise.all(... line with

Promise.mapSeries(batchInputArray, arr => model.predict({ input: arr }))

If you're running this directly in your browser, which it looks like you are, you can enable bluebird Promises simply by including

<script src="https://cdn.jsdelivr.net/bluebird/latest/bluebird.min.js"></script>

[kimamula]

IMO it is difficult to predict that a method like model.predict() has side effects and cannot be run in parallel. Actually I have been struggling to find out why my code is not working correctly for several hours until I come across this thread.

Can this be documented explicitly or is it possible to get rid of the side effect from model.predict()?

[wmathot]

Appears this issue has been open for some time. Just checking back to see if keras-js has been updated to support one or both of the following:

1. Batch prediction (passing multiple feature vectors into model.predict)
2. Parallel prediction (using the same model object in a loop against multiple feature vectors)