How to integrate the gesture recognizer proto in Unity scene?

[B1llyn0m4t35]

Plugin Version or Commit ID

v0.16.1

Unity Version

2021.3.18.f1

Your Host OS

Windows 11 Home

Target Platform

UnityEditor

Description

Hi, I want to build an app to recognize gestures in Unity. Since a script can only be attached to a gameObject if its class is Monobehaviour, how do I use the scripts in the Gesture Recognizer/proto folder to recognize gestures?

Code to Reproduce the issue

No response

Additional Context

No response

[homuler]

I'm not sure what specific issue you're facing, but since GestureRecognizer is not MonoBehaviour, you can instantiate it freely and use it from another MonoBehaviour. For more details, please refer to the tutorial.

[B1llyn0m4t35]

I didn't understand how to create a "GestureRecognizeRunner" script in order to use the Gesture Recognition in my scene, since in all the other scenes, such as "Hand LandMark Detection" or "Face Detection" there is always a Game object named "Solution" with a "Runner" script attached

[homuler]

Have you had a chance to read the tutorial?

In short, without a MonoBehaviour (or a class that inherits from it), code won't run in the scene, so we prepare some MonoBehaviour (just create a new .cs file), attach it to some GameObject, and create an instance of GestureRecognizer within a method related to the lifecycle (e.g. Start). How you use that instance within the MonoBehaviour is entirely up to you.

[B1llyn0m4t35]

Thanks for the tips, I looked at the tutorial and tried it and the gesture recognition works. I have another question though, how do I customize the gestures? Is it possible to add new ones or change the label of the canned ones? I can't seem to find a list of the gestures that I can modify in any script

[homuler]

To customize the model, please refer to the official MediaPipe documentation. I think it should work as is if you replace the model being used.

[B1llyn0m4t35]

Thanks again for your help, I have another question: the model used in this plugin to recognize gestures is a gesture_recognizer.bytes file. If my data are in a csv file how do I convert them in a file format that Unity can access? I tried to open the bytes file but i can't read it

[homuler]

Custom models used with MediaPipe must be in the .task format, which is a model bundle file.

The .bytes file is referred to as a .task file in the MediaPipe repository, and I believe it's a protocol buffer (ExternalFile) data. This repository simply uses the files provided by MediaPipe as-is, so for detailed information about the file format, please ask in the MediaPipe community.

That said, I believe the method to handle it is described in the link I shared earlier (e.g. export_model).

See also https://ai.google.dev/edge/mediapipe/solutions/customization/gesture_recognizer.