Synthetizing from AMASS dataset

[dmanche03]

I am attempting to utilize the genSynData.py file to synthesize data from the AMASS dataset. However, I've encountered an inconsistency between the official dataset and your code. The Jog_1 example file you're using is in .pkl format, whereas the files provided by AMASS are only available in .npz format. Consequently, I'm facing issues with the code because the load_model function from smpl_webuser.serialization does not support .npz files. I attempted to modify the load_model code, but encountered difficulties with the attributes of the files.

While .npz files contain these attributes: ['trans', 'gender', 'mocap_framerate', 'betas', 'dmpls', 'poses']

.pkl files have these attributes: ['heldout_err', 'resolution', 'all_obs', 'all_tls', 'frame_rate', 'can_mrk', 'poses', 'pose_est_weights', 'gender', 'traindir', 'betas', 'sv_fname', 'trans', 'all_betas']

How can I resolve this compatibility issue between formats, and what preprocessing did you apply to the AMASS data?

[kaufManu]

The AMASS dataset we used at the time was a preview version available in pkl files, hence the loading from pickle files in genSynData.py. It should be pretty simple to adjust the script to load .npz files because you only need 4 things: the gender, the betas, the poses, and the framerate. Assume data_in is the data loaded by the pickle file (following the naming in the script) and data_npz is the data as loaded from the npz file, then the mapping is:

data_in['gender'] -> data_npz['gender']
data_in['betas'] -> data_npz['betas']
data_in['poses'] -> data_npz['poses']
data_in['frame_rate'] -> data_npz['mocap_framerate']

The issues with the load_model function might be a different problem. In this function, the SMPL model is loaded from a pickle file, which should still work. I don't understand why this should even attempt to load a .npz file - only SMPL-X models should be available as npzs. Anyway, this whole script is quite outdated, I would not use the smpl_webuser tool anymore. Instead download the smplx package and follow the instructions on their github repo to prepare the SMPL model files.

[dmanche03]

Thank you for your reply. I'm attempting to synthesize data using the genSynData script, but I have a question. Your paper mentions that the data will be synthesized using the mesh surface, but upon reviewing the code, it seems to me that you're utilizing the body model's vertices (joints) instead. Could you clarify this for me?

[kaufManu]

"Using the mesh surface" and "utilizing the body model's vertices" appears to be equivalent to me. To clarify further, we are using the positions of the vertex IDs stored in VERTEX_IDS to compute the accelerations via finite differences (cf. Ln. 66 which accesses vertex positions that were retrieved on Ln. 95). To compute the synthetic orientations we use the corresponding joint orientation. It would also be possible to compute an orientation from the mesh vertices directly, however the joint orientation is much more well defined and the calibration procedure is designed to align the real sensor to the bone orientation (not some orientation computed on the mesh's surface).

Hope that helps!

[dmanche03]

Hi again,

I'm encountering some issues with my projec. After obtaining accelerations and orientations by synthesizing all available data, and integrating them into a custom recurrent neural network, I noticed the synthesized data did not seem accurate.

To investigate the source of the problem, I compared my synthesized data against the data from your Synthetic_60FPS.zip file, specifically examining the IMUs that shared vertex IDs between my data and yours. I observed significant discrepancies between the two.

Could you please confirm if the data within Synthetic_60FPS.zip has undergone any normalization or preprocessing steps?

Thanks for your assistance.

[kaufManu]

Hi, as far as I remember Synthetic_60FPS.zip has not undergone any normalization or preprocessing steps. What is the discrepancy that you observe?

[dmanche03]

To obtain data for training a network, I have used the synthesis code you provided, modifying some vertex IDs VERTEX_IDS = [6740,5763,4560,5213,4361,6060,3021,3498] corresponding to rfoot, lfoot, rlowleg, llowleg, rhighleg, lhighleg, back (root), chest; compared to the original VERTEX_IDS [1962, 5431, 1096, 4583, 412, 3021]), and also modifying the following lines of code:

for a_global in A_global_list:
        ori_left_arm = a_global[18][:3, :3].r
        ori_right_arm = a_global[19][:3, :3].r
        ori_left_leg = a_global[4][:3, :3].r
        ori_right_leg = a_global[5][:3, :3].r
        ori_head = a_global[15][:3, :3].r
        ori_root = a_global[0][:3, :3].r

        ori_tmp = []
        ori_tmp.append(ori_left_arm)
        ori_tmp.append(ori_right_arm)
        ori_tmp.append(ori_left_leg)
        ori_tmp.append(ori_right_leg)
        ori_tmp.append(ori_head)
        ori_tmp.append(ori_root)

        orientation.append(np.array(ori_tmp))

by:

for a_global in A_global_list:
        ori_left_foot = a_global[10][:3, :3].r
        ori_right_foot = a_global[11][:3, :3].r
        ori_left_lowleg = a_global[4][:3, :3].r
        ori_right_lowleg = a_global[5][:3, :3].r
        ori_left_highleg = a_global[1][:3, :3].r
        ori_right_highleg = a_global[2][:3, :3].r
        ori_chest = a_global[9][:3, :3].r
        ori_root = a_global[0][:3, :3].r

        ori_tmp = []
        ori_tmp.append(ori_left_foot)
        ori_tmp.append(ori_right_foot)
        ori_tmp.append(ori_left_lowleg)
        ori_tmp.append(ori_right_lowleg)
        ori_tmp.append(ori_left_highleg)
        ori_tmp.append(ori_right_highleg)
        ori_tmp.append(ori_chest)
        ori_tmp.append(ori_root)

        orientation.append(np.array(ori_tmp))

By comparing the synthesized data using vertex 3021 (which corresponds to the lower back and is shared in both pieces of code), it is possible to see that something has happened causing the same result not to be obtained. I am attaching the following images corresponding to your synthesis Synthetic_60FPS\AMASS_ACCAD\Female1General_c3dA1_SB__SB2__SB_Stand_dynamics.pkl, and the synthesis I made using my code on the same file from AMASS:

At first, I thought it might be due to noise or a scaling error, but after comparing more files, I realized that there were syntheses that had nothing to do with each other.

I'm a bit lost as I can't understand where the error is. Thank you very much for your help.