ROMP icon indicating copy to clipboard operation
ROMP copied to clipboard
verified publisher icon Arthur151

Data Used in Protocol 1, 2, and 3

Open mkhoshle opened this issue 2 years ago • 3 comments

@Arthur151 Hi, I have been looking into the Tables on comparisons across various models. There are several protocols.

  • Could you please help me understand the specific datasets used for training and testing in each protocol and what are the differences between them?

  • based on my understanding protocol2 use 3DPW during train and testing (with *) or only uses it during testing. Is this correct?

  • Are protocols 1 and 2 the same except that protocol 1 does not use ground truth during inference? I am a bit confused about this.

Thanks in advance,

mkhoshle avatar Feb 11 '23 03:02 mkhoshle

Hi Mahzad Khoshlessan,

If you are interested in the evaluation protocols on 3DPW used in ROMP, I think I can help.

The protocol 1 is borrowed from 3DPW challenge, ECCV 2020. We don’t use any data from 3DPW for training. We directly evaluate on the entire set (training, valuation, and test set) of 3DPW. Different from other single-person method that need to compute the bounding box using the ground truth 2D poses, ROMP don’t use any ground truth during inference. Especially, we use the evaluation code and ground truth provided by 3DPW challenge for evaluation, which is different from commonly used ground truth from VIBE.

The protocol 2 evaluate only on the test set. And the ground truth and evaluation code are brought from VIBE. Similarly, ROMP doesn’t use any ground truth during inference.

Best, Yu

Arthur151 avatar Feb 11 '23 07:02 Arthur151

Hi @Arthur151 ,

Thanks for the feedback. I see. So the datasets that are used across protocols 1 and 2 during the training part are the same. Is this correct?

  • Training datasets in protocols 1 and 2 are: 3d pose [h36m, mpiinf], pseudo-label 3D dataset [UP], 2d pose [coco, lsp, mpii, AICH]. Is this correct? What is the percentage of each data used during training? In the .yml file we need to provide the percentage of each data that we want to use and I am not sure what they need to be for both protocols 1 and 2.

  • Where can I find the evaluation code for 3DPW? I could not find anything on the challenge page. Is this the same evaluation code available in the ROMP repository? https://competitions.codalab.org/competitions/24938#results

Thanks in advance,

mkhoshle avatar Feb 12 '23 16:02 mkhoshle

@Arthur151 I would really appreciate it if you could answer the above questions.

Thanks in advance,

mkhoshle avatar Feb 25 '23 23:02 mkhoshle