Feature Request: Process video using the Kinect body tracker on image/depth sequences

[aiwhoo]

I have seen the record playback feature. However, could we add a feature To utilize the body tracker from a different data source —-particularly, a sequence of RGBD images (either pngs or open3d objects). Intrinsics could be passed into the camera configuration.

[ibaiGorordo]

Hi,

Interesting idea. Do you mean the data from another type of RGBD camera, or images from the Kimect but saved into another format?

In theory it might be possible, but I will have to try it because there might be some limitations. Do you have some example data? However, it will probably take time.

Ibai

[aiwhoo]

Yes, capturing data from another type of RGBD camera (e.g. custom built, or even other commercially available cameras). Honestly, the kinect has the most stable/accurate algorithm that we've come across.

I've created a google drive with sample RGBD images. Please ignore the Christmas pajamas. https://drive.google.com/drive/folders/1rdqPBk6rS5H3ekkKGECxK5qKg1-ll2DA?usp=sharing

The depth dimensions are 192 x 256 and the RGB dimensions are 1920 × 1440.

The filename format is <frame>_<timestamp>.jpg and <frame>_<timestamp>_depth.txt

camera.txt contains the intrinsics matrix in the following format

fx,0,ox 0,fy,oy 0,0,1

The values fx and fy are the pixel focal length. The values ox and oy are the offsets of the principal point from the top-left corner of the image frame. All values are expressed in pixels.

We've essentially loaded in the images into open3d to create a <class 'open3d.cpu.pybind.geometry.RGBDImage'> but haven't found a nice way to communicate with the kinectazure library.

import open3d as o3d
import numpy as np

frame_depth_map = np.loadtxt(open("example_depth.txt", 'rb'), delimiter=",", skiprows=0, dtype=float)

def get_rgbd_img(color_img_path, depth_img_path, frame_num):
    color_raw = o3d.io.read_image(color_img_path)
    depth_raw = o3d.io.read_image(depth_img_path)
    rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(color_raw, depth_raw)

[mrcfschr]

Has anyone had any success with this? Any good leads on how to best accomplish this? Thank you!

[ibaiGorordo]

Not yet, I have been busy with other projects. It is an interesting idea, but to be honest it is probably quite hard to get it working (with a lot of debugging what the SDK does in the background), so I am not sure when I will give it a look.

[peymoon]

following up on this, is there anyway to get body tracker frame by feeding depth (and RGB) frames to the tracker (rather than passing the capture to the bodyTracker.update function)?

[ibaiGorordo]

I tried to debug the ONNX model a while ago, but ended giving up. If I remember correctly, I think the model did not output the 3D pose, so that part of extracting the 3D pose from the 2D pose and depth is probably done inside the library.

And it might be possible to convert the RGBD image into the kinect format with the raw data buffer, but since the camera properties is going to be different, it will probably not work properly.