issue about the evaluation code

[suhaisheng]

Hello, I observe that when using the evaluation code from the official ActivityNet repo, the performance can be higher than the one from the official THUMOS14 repo nearly 2% under iou=0.5 and 6% under iou=0.1. I really think it maybe unfair to some extent for comparison with other sota...

[bellos1203]

Hello, I'm not sure why the authors used the ActivityNet version, I just followed the method in the paper.

However, when I used the THUMOS14 eval code, it's not that low as you mentioned. I got the following result. 0.483 | 0.431 | 0.354 | 0.273 | 0.199 | 0.123 | 0.071 | 0.032 | 0.005 The above result was calculated using the checkpoint I uploaded. I didn't report it in the README, but the number is lower than the paper only in the case of the first three criteria. Actually, the performance is higher on higher tIoU.

Thanks :)

[suhaisheng]

Thanks for ur reply! However, when I used the detection results (txt file, which is transferred from the json file u uploaded), I got the detection performance listed as follows. 0.465688 | 0.402861 | 0.324524| 0.250325 | 0.171563 | 0.097312 | 0.047556 | 0.012638 | 0.000900 results.txt Maybe there is something different about the evaluation code. Can u share the evaluation code from the official THUMOS14 repo which u used? Thx!

[bellos1203]

Well, I didn't change the official THUMOS14 eval code a bit. The one thing I modified was to remove the 000270 video, which is falsely annotated, in the ground truth file. Maybe that's the reason for the difference, but I don't think it would make such a difference, though. I updated my code to generate the txt file. Now you can test that file by using the Matlab code.

Thanks :)

[suhaisheng]

I download ur uploaded txt file and test it using the THUMOS14 eval code, and I got the same results as i listed before.. Do u use the same eval code downloaded from this link: http://crcv.ucf.edu/THUMOS14/download.html and test the result using TH14evalDet.m script. If so, the results should be the same as mine... Thanks anyway!

[bellos1203]

Oh, I didn't know that there is another version of eval code. I used http://crcv.ucf.edu/THUMOS14/eval_kit/THUMOS14_evalkit_20140818.zip, which introduced in http://crcv.ucf.edu/THUMOS14/THUMOS14_Evaluation.pdf. It seems your version is newer, so then your result would be correct. Sorry for the confusion.

And for the opinion about the sota, I think it's quite fair even for the results from THUMOS eval code. Because the paper appeared in CVPR 2018, the UntrimmedNet was sota until then. The STPN is still outperforming the UntrimmedNet with that result.

Thanks :)

[bellos1203]

In addition, I got the following results with your code. 0.470 | 0.407 | 0.329 | 0.255 | 0.175 | 0.100 | 0.049 | 0.013 | 0.001 Now it's similar, and I think the difference comes from the excluded video.

Thanks :)