WTAL-Uncertainty-Modeling icon indicating copy to clipboard operation
WTAL-Uncertainty-Modeling copied to clipboard

Published 20 hours ago • verified publisher icon Pilhyeon

Reproducing the results on ActivityNet

Open tobbysong opened this issue 4 years ago • 13 comments

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

  • [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

  • [num_segments: 50] (0.0275, 0.31)
    • [class_th: 0.1] (0.0332, 0.36)
    • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

tobbysong avatar Oct 04 '20 04:10 tobbysong

First of all, I apologize for the missing details about the experimental settings on ActivityNet. Since I extracted the features for ActivityNet by myself, they might be different from those you used. For example, the sampling rate of the videos or the pre-trained I3D models could be different. I used the original fps of the videos and the pre-trained models from this repo. For the hyperparameters, the batch size, the margin, alpha, (r_act, r_bkg), and the NMS threshold are set to 512, 250, 1e-4, (5, 50), and 0.8, respectively. I hope this would help.

Pilhyeon avatar Oct 12 '20 06:10 Pilhyeon

Can you share the feature of act1.2?

xumh-9 avatar Oct 18 '20 03:10 xumh-9

Thanks for your wonderful work. Could you please share your features in Activitynet 1.3?

cecilia930426 avatar Oct 21 '20 03:10 cecilia930426

Hi, have you got the better result after you set the hyperparameters as mentioned? Thanks.

P-DX avatar Dec 24 '20 02:12 P-DX

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

  • [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

  • [num_segments: 50] (0.0275, 0.31)

    • [class_th: 0.1] (0.0332, 0.36)
    • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

Hi toby ,

Can you kindly share your version of the repo for ActivityNet dataset in this email : [email protected] if possible ?

sauradip avatar Dec 28 '20 23:12 sauradip

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

  • [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

  • [num_segments: 50] (0.0275, 0.31)

    • [class_th: 0.1] (0.0332, 0.36)
    • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

hi @tobbysong, have you got the better result after you set the hyperparameters as mentioned?

liming-ai avatar Jan 25 '21 08:01 liming-ai

@xumh-9 @cecilia930426 @sauradip ActivityNet1.2 from W-TALC github. ActivityNet1.3 from A2CL-PT github.

kiwiJ avatar Mar 07 '21 11:03 kiwiJ

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful. @Pilhyeon

hawklucky avatar Mar 24 '21 15:03 hawklucky

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

liming-ai avatar Mar 25 '21 02:03 liming-ai

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

hawklucky avatar Mar 25 '21 02:03 hawklucky

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

liming-ai avatar Mar 25 '21 02:03 liming-ai

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

Richard-61 avatar Dec 02 '21 20:12 Richard-61

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

Me too. Has anyone reproduced the reported performance?

Jockey721 avatar Apr 05 '22 03:04 Jockey721