Multiple Palm detections by inferencing the model

[Constannnnnt]

Describe the expected behavior:

I intended to use Mediapipe in Unity (where mediapipe is not supported I think) and since its Barracuda engine only supports ONNX format, so I decided to convert the palm_detection.tflite into ONNX format. Before that, I would like to see the model outputs first so I can better understand and process the returned results. So, I ran a test (I know it is a bit weird to test in python when I want to build Unity in the end, but u know, python is fast for testing....)

Here is a code block of my test in Jupyter notebook:

!python -m tf2onnx.convert --tflite palm_detection.tflite --output palm_detection.onnx

import cv2
import numpy as np
import onnxruntime as rt

img = cv2.imread('test_image.jpg')
dim = (128, 128)
img = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
tensor = (img / 127.5 - 1.0).reshape((1, 128, 128, 3))
sess = rt.InferenceSession('palm_detection.onnx')
pred = sess.run([], {'input': tensor.astype(np.float32)})

###some drawing stuffs

The test_image.jpg has two hands, so it is expected to detect the palms of both hands. However, it only detects the left hand. I don't think there is anything wrong with my drawing functions because all predicted offsets in the returned 1 x 896 x 18 tensor are processed and the left palm is correctly detected.

Therefore, I am wondering if the tflite model mentioned above is able to detect multiple palms. If yes, how can I achieve detecting multiple palms. If no, is there any other solution to this question?

Thanks in advance.

[brucechou1983]

If you read the palm_detection_gpu.pbtxt in the palm detection module, you'll see many calculators other than tflite inference calculator. Especially there is a non max suppression there. What's the point of NMS if the model can't detect multiple hands?

Base on my model conversion experience, either the onnx file has some subtle difference (say op implementation or default values) from the tflite, or the post processing is different.

[MJ1998]

@Constannnnnt Assuming you are doing all the post processing(evaluating scores, nms, rotation, derotation) correctly, see if you are not changing the original image after any of the post processing step. also you can check if there are atleast 2 scores with value above 0 then you can assume model is detectng 2 hands correctly and most likely your post processing has some issue!

[aa12356jm]

@Constannnnnt hi，can you share the complete python codes for palm detection？thanks

[lucasjinreal]

I can't convert the tflite to onnx:

tflite2onnx/layout.py", line 16, in <listcomp>
    transfrom_axis = [input[p] for p in perm]
IndexError: list index out of range

any ideas?

[Brightchu]

I can't convert the tflite to onnx:

tflite2onnx/layout.py", line 16, in <listcomp>
    transfrom_axis = [input[p] for p in perm]
IndexError: list index out of range

any ideas? @jinfagang use tf2onnx instead of tflit2onnx. and look out for the env settings. should be python3.7+ and tensorflow 2.x.

[kuaashish]

Hello @Constannnnnt, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.