offline processor produces different body tracking results when running multiple times on same video

[l1ndaB]

Describe the bug We used the offline processor multiple times on a mkv-file and got different results. But we expect to reproduce the same skeleton every time we run the body tracking.

To Reproduce Take the same mkv-file and run the offline processor on it for multiple times. If you compare the different outputs, some are the same, but some have different results.

Expected behavior We expect the same output every time we run the body tracking on a mkv-file.

Screenshots We get different outputs when running the offline processor multiple times on the same video file. In the image you can see the joint positions found in the exactly same frame (and same person) in different results all created by the offline processor.

Desktop (please complete the following information):

• OS with Version: Windows Version 21H1
• SDK Version: Body Tracking SDK Version 1.1.1 and 1.1.2 both show this behavior
• Firmware version: The Video got recorded with RGB camera firmware 1.6.110 and depth camera firmware 1.6.8

Recording settings: The Video got recorded with the following settings: FPS: 30 Depth Mode: NFOV_2X2BINNED Color Format: MJPG Color Resolution: 1536p IMU enabled: Yes Sync Mode: Master (we got two cameras running on the same time)

[qm13]

This is expected as the neural network is seeded with random weightings on initialization.

[l1ndaB]

@qm13 Thank you very much for your quick answer. We have some follow up questions:

1. Is it possible to change this random weightig to a fixed weight in the tracker configurations?
2. Does this behavior occur in every version of the body tracking sdk or only in specific versions?
3. Do you have some parameters about the differences that occur because of these random weights? (Like the maximum difference of a position)
4. Just for curiosity: Why is the neural network seeded with random weightings?

[Aquariusquintini]

@qm13 qm13: I have also noticed that the body tracking results vary every time I run it on the same video.

I have tested this a little further and noticed that it is only occurring when the CUDA is used as processing mode, with DirectML or CPU the body tracking results are the same accross multiple runs. Could it be a bug in the CUDA processing?

Steps for reproduction: Take a video (we used a 1 minute video of a mannequin). For x = 1; x<10; x++

Process the video with body tracking using

• CPU
• CUDA
• DirectML end

Compare the results of the body tracking coordinates.

Expected behavior: Result of the body tracking is the same for every run

Observed behavior: Using CUDA there are differences in joint possitions of up to 65 mm! CPU or DirectML do not have any difference accross multiple runs.

[qm13]

@Aquariusquintini most interesting results. Are you seeing the same sort of variation with TensorRT or just CUDA?

[Aquariusquintini]

@qm13 we did not try TensorRT yet. If it helps, I could give it a try next week and get back to you with the results.

We have also first indications that running body tracking in CPU or DirectML mode on the same video on different machines results in differences comparable to the between run results in CUDA. However, we are still investigating this.

[Aquariusquintini]

@qm13 I have done some tests and can now confirm that TensorRT has similar behaviour compared to CUDA, also it produces different results on different runs.

I can also confirm that running body tracking on different machines (using the same body tracking SDK and the same video) produces different results in all processing modes. Initial results show that for Pelvis they are in the range of about 2-4 mm, however for the LeftFoot for example up to 80 mm!

[l1ndaB]

We have further analyzed the problem and published a paper on it, which can be found here: https://www.mdpi.com/1424-8220/23/2/878