Depth error on elbow joints in pose model

[BitLoose]

Whilst depths for the pose landmarks are generally correct, the elbows are always in the depth foreground. I enabled depth rendering for all joints and lines, and this can be clearly seen. I've searched for this issue and found nothing, but this must be a known problem, no? The visibility float is also always under 0.24 for elbows even when clearly identified and marked, whilst other body landmarks come back as +0.8. Do you have a solution?

[sureshdagooglecom]

Hi @BitLoose , Could you please confirm in which platfrom you trying this solution?

[BitLoose]

Hi, thanks very much for the reply.  I am using Windows 10, latest service packs,  with Visual Studio 2022 C++ and I've tried with both 0.8.10.0 and 0.8.10.2, or is that 0.10.8.2? Anyway, I used the very latest mediapiple off github, and the previous version.

The elbow joint off the pose model is always with Z in the foreground, I enabled Z rendering in the .pbtx, and also checked the Z values. Really looks like a bug.

I'm also trying to add the model_complexity and smoothing side packets in the main code, c++, to the pose model but although the holisitic .pbtx file has the packets defined, the graph is unable to find them defined.

I just cannot find out how to do this, and the only option is to completely debug the entire mediapipe structure, unless some kind expert would like to help.

Rob

On 02/08/2022 15:08, Sureshda wrote:

Hi @BitLoose https://github.com/BitLoose , Could you please confirm in which platfrom you trying this solution?

— Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/3555#issuecomment-1202652370, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYZGD7JOKYCPA75FN2AGA3VXETUXANCNFSM54ZQU76A. You are receiving this because you were mentioned.Message ID: @.***>

[schmidt-sebastian]

@gkarpiak Do you know the answer to this question?

[hadon]

The developers of the mediapipe Pose model, including https://github.com/ivan-grishchenko, may be interested to learn about your observations, that the elbow landmarks are never reported with negative depth or with high visibility.

About the "model_complexity" and "smooth_landmarks" side packets, side packets are normally passed from the application to the CalculatorGraph through the CalculatorGraph::StartRun API. It is also possible to provide values for certain side packets inside the CalculatorGraphConfig using a calculator such as "ConstantSidePacketCalculator". An "output_side_packet" from "ConstantSidePacketCalculator" can be used as the "input_side_packet" of the "PoseLandmarkCpu" subgraph (from pose_landmark_cpu.pbtxt) or the "PoseLandmarkModelLoader" subgraph (from "pose_landmark_model_loader.pbtxt").

[BitLoose]

All I can report is that the depth for the elbows is way too foreground, I used the last two releases of MP and both show the same results.

In regards to the side packets et al, as there is not much documentation, without examples of the protocols required, its very difficult for a non-team coder to discern what's required. An example showing how to set the complexity and smoothing would save scores of people many hours.  I've already gone through the code and see that smoothing is enabled by default using the one euro algorithm, took some time to trace it through, but model complexity I have yet to locate.

If the aim was to obfuscate the code, making it difficult to use, then its a job well done. Sorry for the sarcasm, but sometimes someone has to say it as it is, IMHO.

On 07/09/2022 10:28, Hadon Nash wrote:

The developers of the mediapipe Pose model, including https://github.com/ivan-grishchenko, may be interested to learn about your observations, that the elbow landmarks are never reported with negative depth or with high visibility.

About the "model_complexity" and "smooth_landmarks" side packets, side packets are normally passed from the application to the CalculatorGraph through the CalculatorGraph::StartRun API. It is also possible to provide values for certain side packets inside the CalculatorGraphConfig using a calculator such as "ConstantSidePacketCalculator". An "output_side_packet" from "ConstantSidePacketCalculator" can be used as the "input_side_packet" of the "PoseLandmarkCpu" subgraph (from pose_landmark_cpu.pbtxt) or the "PoseLandmarkModelLoader" subgraph (from "pose_landmark_model_loader.pbtxt").

— Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/3555#issuecomment-1239142014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYZGD5TLBS6YHXXSWM3IALV5BN5HANCNFSM54ZQU76A. You are receiving this because you were mentioned.Message ID: @.***>

[BitLoose]

I tested the elbow depth error with MP release 0.8.11 and the error persists. The elbows are nearly always in the far foreground whatever the pose. How is depth calculated, is it a post-process after the model, if so, where are the files located, has anyone verified this bug yet, or is it known and abandoned?

[BitLoose]

which questions?  The reason why depth on the elbows is too far in the foreground?  Can't see why the CNN would just fail on the elbows, so probably some sort of post processing coding issue.

Sure the engineers that put it together already know the answer as the fail is very obvious.

On 06/09/2022 20:46, Sebastian Schmidt wrote:

@gkarpiak https://github.com/gkarpiak Do you know the answer to this question?

— Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/3555#issuecomment-1238581649, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYZGDYOZJAH7HHGG2CK533V46NQ7ANCNFSM54ZQU76A. You are receiving this because you were mentioned.Message ID: @.***>

[kuaashish]

Hello @BitLoose, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No