SmoothNet
SmoothNet copied to clipboard
cure-lab
Please update config file
None of these config file can using for loading according pretrained models.
A lot of mismatch....
Hi, @jinfagang ! Thank you for your focus! We didn't see such a mismatch in our code environment. Would you provide us with more detail for your question?
@juxuan27 Hi, main 2 question;
- what's diference of SMPL and 3DPose? their config are same, but definitely the pretrained model are different (file size not same);
3dkeypoint paree ->
smple:
- using above config:
MODEL :
SLIDE_WINDOW_SIZE : 100
HIDDEN_SIZE : 512
RES_HIDDEN_SIZE : 16
NUM_BLOCK : 1
DROPOUT : 0.5
but pretained claimed need 32 input size....
As shown in the training part, you need to assign a window size to train the SmoothNet. To test a SmoothNet, you must set the corresponding window size.
python train_smoothnet.py --cfg [config file] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size]
@ailingzengzzz thank u . But still I didn't get my puzzle solved, I am try using on hybrIK , which model should I use? 3d keypoints or SMPL? Essentially, the model also outputs a 24x3x3 like 3d keypoints.
Hi @jinfagang,
Actually, Hybrik is a model to estimate SMPL. The performance of Hybrik depends on the train and test sets. If your data is fast-moving and with high errors (over 80mm), you may try [AIST-VIBE-3D]; If the data is slow-moving and with low errors (around 50mm), you may try [H36M-FCN-3D]. Generally speaking, you can first try [H36M-FCN-3D]. We discuss the generalization ability of SmoothNet in Section 2.5 in the Appendix (https://arxiv.org/pdf/2112.13715.pdf) and Table 7,8,9.
In our experiments, SmoothNet is trained 3d position modalities can be better than SMPL 6D rotation matrix. Thus, you can use the pre-trained 3d positions model to smooth the 6d rotation matrix.
Please make sure the input size of SmoothNet is [N, T, C], where T is the window size, and C is all spatial dimensions (e.g., 24*6). After smoothing, you can reshape the [N, T, C] into [N, T, 24, 6] or [N, T, 24, 3, 3].
@ailingzengzzz thank u. Hyprik trained on 3dpw, does h36m-fcn-3d works better?
I am still confused, there are so many combinations, you mean your model can fit both [N, C, T] where C is 3x3 or 6 simuteneously?
which config need N, 3x3, T and which need N,6,T?
Yes, SmoothNet is a temporal-only model, which shares weights across spatial dimensions (any dimensions can be smoothed in parallel). There is no need to set the config of spatial dimensions.
@ailingzengzzz thanks I get it. I would try send my rotmat3x3 directly into model to see any improvements. Even though send a rot6d would be better.
By theway, as for the camera transition smooth, Does smooth net can handle it? It seems many pose3d model predict cam trans standalone.
Hi @jinfagang,
We have not tested SmoothNet for the camera transition. Could you please try it on the rotation matrix and give us feedback?