Training on Human3.6M dataset

[hliuav]

Can I use this code to train on human3.6M dataset(16 landmarks) by just simply replace the dataset and partition.txt since the results I get look not as good as those in the paper

[YutingZhang]

For Human3.6M, we also used optical flow as self-supervision (see Appendix C). The result may not be as good as those in the paper if the optical flow is not used. The implementation of the optical-flow based loss is already implemented at https://github.com/YutingZhang/lmdis-rep/blob/d2292d97bef5d278cb1f4663ef7785f0621212af/net_modules/auto_struct/keypoint_encoder.py#L426-L506

However, we have not released the code for the data loading and (OpenCV based) optical flow computation for Human3.6M. We plan to do that soon.

[hliuav]

Thank you for your quick reply. I also find that if the network are trained with pictures with background, the landmarks tend to form a circle and each landmark only varies a little in its local region. Most of the cases shown in the paper are also trained with the images of similar pose(car, animals etc.) Only human3.6M dataset has various of poses. Is that the reason why we need to extract the background of the human3.6M dataset, that is, to make sure the network won't learn landmarks from background?(I have tried to train with human3.6M dataset with background, the network almost learn nothing)

[YutingZhang]

Sorry for the delayed response due to my recent job transition. The method is not robust to background variations for human body images (though it works for faces).
The human body is more complicated than other objects regarding the pose variation and the viewpoint of interest. So the foreground object structure is also harder to capture. I think that is why an easier background is needed.

[ender1001]

Thank you for your great job. It really helps me a lot in my current work. I have encountered a similar problem in the background. Actually, I have extracted only the foreground from a video, but the method still recognized part of the foreground as background, hence missing some important landmarks. I am wondering if I can turn off the background channel in both encoding and decoding. I found some related options in your code but failed to enable them. Do you have any suggestions? Thank you.

[jojolee123]

Thank you for you nice work! Can you provide a download link of Simplified Human3.6M dataset & Human3.6M dataset? Waiting for your relay, thanks！

[YutingZhang]

Just added a Google Drive link in the readmo: https://github.com/YutingZhang/lmdis-rep/ Thanks!

On Wed, Jul 20, 2022 at 10:05 PM jojolee123 @.***> wrote:

Thank you for you nice work! Can you provide a download link of Simplified Human3.6M dataset & Human3.6M dataset? Waiting for your relay, thanks！

— Reply to this email directly, view it on GitHub https://github.com/YutingZhang/lmdis-rep/issues/2#issuecomment-1190953838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4AML7S7EFZOPZ444E2ZETVVCWADANCNFSM4FO64RBA . You are receiving this because you commented.Message ID: @.***>