Evaluation Results with ICCV'21 Paper

[ludy3996]

Hi, thanks for your great work on ICCV'21. I have some questions and issues with your code. I just clone your code (didn't modify official code) and evaluated the provided pre-trained model with these codes. But I got so different results with your ICCV paper, only got the evaluation result like this.

Test APE [4.190873301445142, 15.652097037362648, 14.476590518962245, 17.752781245065474, 14.795194316051749, 13.24986283658489, 14.723207273690173, 14.2541448259369, 13.721858107615398, 13.556832176482954, 13.431385539737901, 13.9537314120196, 13.902865999886918, 12.268731942753899, 13.682642174726016, 13.855254758015043, 14.981251266174741, 12.796628382808017, 14.758570286169096, 8.079425356527656]

Test AVE [10.178670862604642, 15.219314325004246, 12.879057965257015, 14.139660217758474, 13.689597073921833, 8.454485149774731, 9.164908408357755, 6.780733172433479, 9.70430640765224, 7.1300269325197965, 5.382036185673594, 20.322144246346795, 13.266659138706576, 8.909401371480373, 12.573738910250547, 20.25171573373726, 14.21424176749085, 10.286702136810037, 13.424342641988945, 5.88676996947937]

Test CEE 1.2373948167908533

Test SEE 0.46560313116859303

I wanna reproduce the performance of ICCV'21 with your official model and code. But I don't know provided pre-trained is the best model or not, and how to modify this code to get it. Do I need to re-train the model or do you have any idea about this? And another issue is when using python dataProcessing/meanVariance.py -mask '[0]' -feats_kind rifke -dataset KITMocap -path2data ../dataset/kit-mocap -f_new 8 to calculate the mean and variance for Z-Normalization, got some errors about the length of mask. Then I used this one python dataProcessing/meanVariance.py -feats_kind rifke -dataset KITMocap -path2data ../dataset/kit-mocap -f_new 8 (I saw code from dataUtils.py and there is some defaults with mask=[0], but I don't know if this is correct.

Thanks,

[anindita127]

Hi,

Thanks for your interest in our work. Recently the repository went through some updates. Can you check in once more and see if your issues sill persist.

[HappyPiepie]

Hi , thanks for your work for this field. I hava some problems when i run your eval coda. I got APE_LOSS： [1.6152539077560253, 3.8035130007033446, 3.801077495838645, 4.009686965172017, 3.7679262980997006, 3.590032654908319, 4.529732276969545, 4.89012817142057, 3.574709865812325, 4.043609161419107, 4.578285699350569, 3.8500040182153494, 4.156827169357215, 3.902301512267607, 4.1243365680896416, 3.839924943096757, 4.410920671962248, 3.9593780655352617, 4.329834376329709, 3.3412870973680375] is close your paper's results.

And , AVE_loss:
[0.44463602033627375, 0.7030023537442466, 0.7308105546397969, 0.805257988857508, 0.7744137068526745, 0.40802066304414186, 0.6775546575277528, 0.6388468728953295, 0.5033926311021557, 0.38493773164501277, 0.3819525868883675, 0.6934178872048002, 0.5638059260649851, 0.3720031997450148, 0.5573624620957186, 0.674750450305267, 0.6029626064853397, 0.4764308816370605, 0.6281975160633145, 0.4648465112427174]

test_loss_cee: 0.13350790943356924 test_loss_see: 0.047305966971594324 is so different with your paper. Can you help me ?

[anindita127]

Hi, thanks for your interest in this work. Have you checked if all the sequences in the test set is the same? I have received a feedback saying the dataloader of the code was loading a lesser number of test data samples than what was originally noted in the paper. Please check into that!

[HappyPiepie]

Hi,thank you for your reply. I 'm interested your reserach. But i check the number of test is 1272. However the result AVE is so diffurent from paper. Can you help me？ **************************ntest = 1272 AVE_loss: [0.11498997069991236, 0.36148732210997464, 0.331772028897207, 0.3569755042491488, 0.3645807766719102, 0.348606916867408, 0.4495298685360269, 0.36152220935468593, 0.36813386458562714, 0.31260547344663797, 0.27812345266831623, 0.4559474845800431, 0.4220093915709082, 0.39660299227905194, 0.43399406737483454, 0.4691431327129764, 0.39240389395484915, 0.48269123247404677, 0.4334432511608902, 0.44937287047752356] (text2motion) beibeigh@beibeigh-System-Product-Name:~/Research/Projects/LanguageText2Pose/Complextext2animation-main/src$