Body index map is very inaccurate in many scenarios

[cavisions]

Describe the bug

In many scenarios the body index map is very inaccurate. It has holes. Sometimes some body parts disappear completely. Sometimes the map is the sum of 2 users even when one user is standing about 1 meter behind the first one.

To Reproduce

This problem can be observed in the viewer application delivered with the body tracking SDK.

Expected behavior

I'd expect to have the body index map to be more accurate what will (in my opinion) influence also the quality of body joints placement. I'd expect it to work at least as good as it was working on Kinect v2.

Screenshots

Desktop (please complete the following information):

• OS with Version: 1903
• SDK Version: 1.3.0
• Body Tracking SDK Version: 1.0
• Firmware version: 1.6

[amantley]

I am seeing something similar, here is a video that shows the shoulders disappearing when you are wearing a black t-shirt. if the same person puts on a blue coat then the appear correctly.

https://microsoft-my.sharepoint.com/:v:/p/anantley/Ect0R3LiIRpJsz_oA3Z2VcEBfPsZSi2LnUawof66b79Qsw?e=eLtEyz

[cavisions]

In my case it doesn't depend on what I'm wearing - the same effect is observed in many situations. It is also not dependent on the lighting conditions. It looks like it is so far poorly written. In the same conditions the Kinect v2 sensor is doing the job perfectly.

[cavisions]

It looks like the problem is not at all addressed in the Body Tracking SDK Version 1.0 what is very problematic. Do you have any plans to work on this issue?

[eshirima]

@cavisions hey, I was wondering if you could help me with this question since you seem to have it done.

[fractalfantasy]

@cavisions have a look at this feature request to implement kinect v2 ‘random forest’ model as an option - kinect v2 body tracking generated better body index, frame rate, latency and accuracy for dynamic movement:

https://feedback.azure.com/forums/920053-azure-kinect-dk/suggestions/39945454-legacy-body-tracking-like-kinect-v2

[kinetronix]

I've had some success in using a simple algorithm/heuristics for extracting an accurate the body index map from the current depth map and the last known body index map -- essentially start from body sdk-provided body index map and remove some unlikely pixels then spread the other pixels around the object unless close to the floor -- but it's not very robust so I have to limit how the user must position himself to interact. The advantage of k4a over Kinect V2 is the depth map is delivered faster though.