Output not update with customized model

[lookoutking]

Environment (please complete the following information):

• OS/OS Version: both Android 10 and Windows 10
• Source Version: master/01a9965
• Unity Version: 2020.3.20f1

Describe the bug I am trying to add a gesture classification model in the hand tracking scene. I used the tflite model provided in this repo. and implement a new class for gesture classification as the following. The output result did not update after first invoke (results are same with any input value) and there is no error messages show up. Is there anything I miss in the implementation? Thank you for your help.

`public class GestureClassification : IDisposable { private Interpreter interpreter; private float[] inputs; private float[] outputs;

    public GestureClassification(string modelPath)
    {
        var options = new InterpreterOptions()
        {
            threads = 2,
        };

        try
        {
            interpreter = new Interpreter(FileUtil.LoadFile(modelPath), options);    
        }
        catch (System.Exception e)
        {
            interpreter?.Dispose();
            throw e;
        }
        interpreter.LogIOInfo();

        var inputInfo = interpreter.GetInputTensorInfo(0);
        var outputInfo = interpreter.GetOutputTensorInfo(0);
        inputs = new float[inputInfo.shape[1]];
        outputs = new float[outputInfo.shape[1]];
        interpreter.ResizeInputTensor(0, inputInfo.shape);
        interpreter.AllocateTensors();
    }

    public void Dispose()
    {
        interpreter?.Dispose();
    }

    public void Invoke(Vector3[] joints)
    {
        MediaPipeLandmarkToInput(joints);
        interpreter.SetInputTensorData(0, inputs);
        interpreter.Invoke();
        interpreter.GetOutputTensorData(0, outputs);
        Debug.Log(string.Join(", ", outputs));
    }

    private void MediaPipeLandmarkToInput(Vector3[] joints)
    {
        var width = 1280;
        var height = 720;
        var tempList = new List<Vector2>();
        for (var i=0; i<joints.Length; i++)
        {
            var x = Mathf.Min(Convert.ToInt32(joints[i].x * width), width-1);
            var y = Mathf.Min(Convert.ToInt32(joints[i].y * height), height-1);
            tempList.Add(new Vector2(x, y));
        }

        var baseX = 0.0f;
        var baseY = 0.0f;
        float[] tempFloat = new float[inputs.Length];
        for (var i=0; i<tempList.Count; i=i+1)
        {
            if (i == 0)
            {
                baseX = tempList[i].x;
                baseY = tempList[i].y;
            }
            tempFloat[2*i] = tempList[i].x - baseX;
            tempFloat[2*i+1] = tempList[i].y - baseY;
        }
        float max = tempFloat.Select(Math.Abs).Max();
        inputs = tempFloat.Select(x => x/max).ToArray();
    }
}`

To Reproduce Steps to reproduce the behavior:

1. Add the keypoint_classifier.tflite in StreamingAssets
2. Add GestureClassifiaction in HandTrackingSample.cs
3. Invoke with landmark point gestureClassification.Invoke(landmarkResult.joints);

Expected behavior Expected the outputs of keypoint_classifier.tflite varied with different input.

Screenshots Various input value with the same output value

[asus4]

Hi @lookoutking,

I have not tried your model yet. But according to the README in DJI Tello Hand Gesture control, probably you need pre-processing to the input tensors. Please note, that the Vector3 landmarks in the Unity are already normalized in the range [0.0-1.0].

[lookoutking]

Hi @asus4 ,

Thank you for the reply. I have done the normalization in the MediaPipeLandmarkToInput() (in my attached code) and the recognition works fine when I apply the same model with Unity Barracuda. I don't think the problem is about the pre-processing.

[asus4]

@lookoutking you are right, I missed that code. It's weird that the same pre-processing works in Baraccuda. If you still need to run it on TensorFlow Lite, can you upload a reproducible project on Github or somewhere? I will look into it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.