node-tjbotlib icon indicating copy to clipboard operation
node-tjbotlib copied to clipboard

Implement new Rock Paper Scissors recipe

Open modcarroll opened this issue 2 years ago • 5 comments

Watson Visual Recognition is deprecated, so we should find a (preferably) open source replacement

modcarroll avatar Sep 28 '22 20:09 modcarroll

Cesar will own this, he experimented with OpenCV, will train an object detection model for the RPi

jweisz avatar Jan 13 '23 15:01 jweisz

@cmaciel is getting 1.1-1.5 FPS with a pre-trained model (MobileNet, has 90 classes of objects) on a RPi 3. He will re-train the model to focus only on objects found inside a house to get better results.

jweisz avatar Jan 27 '23 15:01 jweisz

Oops, the model was trained but the data labels were missing! :D

jweisz avatar Mar 24 '23 14:03 jweisz

Need to create a list of common household items to train a new model. Maybe focus the model on rock, paper, scissors? Maybe tjbotlib has generic functions for loading local models and running inference, and then we create a separate recipe that uses that API to load a rock/paper/scissors model locally in order to play that game.

jweisz avatar May 05 '23 14:05 jweisz

I propose a two-part solution for the elimination of Watson Visual Recognition:

  1. Remove the see() functionality from tjbotlib and add a look() function which just takes a picture and returns the filename in the filesystem (e.g. somewhere in /tmp). Which is to say, I think this is what takePhoto() already does, so maybe we just rename that function.
  2. Add a new recipe to tjbot for rock-paper-scissors that uses @cmaciel 's on-device vision model. But we likely don't want to bundle the whole model inside the tjbot repo because it's likely big (right?) so maybe the recipe downloads the model from somewhere on first run (e.g. test if it's in the filesystem, if not, then download it). We'd need a permalink for hosting the model somewhere.

@cmaciel if this plan makes sense to you, I'd like to close this card and open two new cards to cover 👆 . Thanks!

jweisz avatar Sep 22 '23 14:09 jweisz