Convolutional-Sequence-to-Sequence-Model-for-Human-Dynamics
Convolutional-Sequence-to-Sequence-Model-for-Human-Dynamics copied to clipboard
chaneyddtt
Using 3D coordinates for joints as input.
I am wondering if this model works for 3D coordinates of joints as input?
I tried to train by changing the load_data in data_utils.py to load 32 joints data (96 dimensions) and also changing the dimensions in human_encoder.py.
I get a weird output as can be seen below: Video
Has someone else tried doing so successfully? Any help would be great. Thanks in advance.
Hi @zaverichintan, I did not try to input only one frame of 3d coordinates before, but i suspect that it would be very hard for the network to learn the motion dynamics with only one frame. You might have to do some modifications to the original network to get it work. It would be challenge I think because one frame can not provide enough information.
Hello @chaneyddtt, I was wondering if you tried giving a sequence of frames (3d coordinates), as for me the network does not seem to learn the motion dynamics.
Can you give more details on how you preprocess the 3d coordinates data?
There is no preprocessing done on the 3D coordinates. I am using mocap library and my code can be found on My commit at the forked repo
I think some preprocess step is needed in your case, such as set the root joint as the original point and normalize the data before feeding into the network.
I perform normalization by setting the root joint at the origin and by rotating each 3D pose in such a way that it faces in the same direction, aligning with the y-axis. I trained the seq2seq model as well as trajectory dependency (https://github.com/wei-mao-2019/LearnTrajDep) with this data and it works for both models without any issues.
I am wondering if the kernel size or the order in which the different joints appear in the vector matter.
Thank you very much for your help!
This paper (https://github.com/wei-mao-2019/LearnTrajDep) also tests our model directly on 3d joint coordinates, can you get similar results as theirs?