[FEATURE] Retrain for offscreen body occlusions

[alexrichardson21]

Is it possible to retrain the network to handle cases where the body is only partially on the screen? (i.e. feet/legs hidden below screen, body running on/off screen on left or right side)

[mkocabas]

Hi @alexrichardson21,

Yes, it is definitely possible. We rely on SPIN pretrained backbone. So, you would need to train SPIN with proper data augmentation. I plan to release an occlusion robust pretrained model in a few days.

[alexrichardson21]

Sounds great! Also is it possible include face landmarks (i.e. DLIB) with SMPL? or would I need to retrain for SMPLX?

[ikvision]

I think https://github.com/facebookresearch/eft can be helpful for occlusions, but it has a slow inference as it Exemplar Fine-Tuning in test time. A short youtube video reviewing the method https://youtu.be/F2_SCM2Oqs4?t=1913

[alexrichardson21]

Found a solution: SPIN doesn't handle offscreen occlusions well but CenterHMR does a much better job of handling occlusions and also yields better results in general. Just swap SPIN with CenterHMR for single frame fitting instead

https://github.com/Arthur151/CenterHMR

[biansy000]

Hi @alexrichardson21,

Yes, it is definitely possible. We rely on SPIN pretrained backbone. So, you would need to train SPIN with proper data augmentation. I plan to release an occlusion robust pretrained model in a few days.

So is the model provided now a occlusion robust one? I still find many unfavorable results after applying the model on the dataset which contains upperbody only ...

[tanmayshankar]

Following up, is there now an occlusion robust model loaded?

[mkocabas]

You can give https://github.com/mkocabas/PARE a try.