Issue in Reproducing the Accuracy

[falguni7]

Hi, I am trying to reproduce the result of the PRP project on UCF-101 dataset split 1 for the c3d model.

Steps followed:

1. python train_predict.py --gpu 0 --epoch 300 --model_name c3d

Best model is at epoch 274 (snippet of last part of output attached that shows the loss and validation accuracy at the 300th epoch)

(Added location of best model - 274 in ft_classfy.py) 2) python ft_classfy.py --gpu 0 --model_name c3d --pre_path 0 --split 1

The best loss and best accuracy model were the same model at epoch 158 and achieved Top-1 accuracy of 100%. Attaching the snippets below.

(Added the best loss model into the test_classify.py file) 3) python test_classify.py

From the output file, I get an accuracy of only 0.335 or 33.5% (expected accuracy of around 69.1%)

Since the accuracy is unexpected, I also downloaded the pretrained model best_model_283.pth.tar from the location: https://onedrive.live.com/?cid=6a4a2ab0702a885f&id=6A4A2AB0702A885F%211366&authkey=%21ALDN0%5FIJt5VWMP8 and performed the steps 2 and 3 on that model. The best loss model after the finetuning gave only 30.5% accuracy here.

Can you please let me know if I have missed any step here or if any of the numbers indicate any issue with training? Thank you in advance.

[yuanyao366]

Is the dataset split uesd in the test_classify.py is also "split1" which is same to that used in ft_classfy.py? And make sure that the fine-tuned model is correctly loaded in the test_classify.py.

[falguni7]

Thank you for the response.

• Is the dataset split uesd in the test_classify.py is also "split1" which is same to that used in ft_classfy.py?

I kept the test_classify.py file unchanged apart from changing the best trained model location in variable "pretrained_path". The Split is 1 according to: https://github.com/yuanyao366/PRP/blob/master/test_classify.py#L107 . Verified the same from the log.

For ft_classfy, I used the command that picks split 1 of UCF dataset: python3 ft_classfy.py --gpu 0 --model_name c3d --pre_path 0 --split 1

Verified that the correct split is picked up from the ft_classfy log too.

• And make sure that the fine-tuned model is correctly loaded in the test_classify.py.

In test_classify.py, I specified the location of the model here: https://github.com/yuanyao366/PRP/blob/master/test_classify.py#L130, which directly picks up the model from the list of models generated during finetuning (in my case, the path being home/Workspace/PRP/outputs/ft_classify_default_UCF-101/01-02-10-17/best_loss_model_158.pth.tar, same folder location as the log from ft_classify output attached below).

Is there any other param to be changed for specifically loading the model? Thank you.

[falguni7]

Hi, just wanted to check if you have any other suggestions regarding reproducing the result. I have taken care to use the correct model loading and splits. Did you face this issue of low accuracy at any stage? Also, I know the project is somewhat older but do you happen to remember if the results obtained in the paper were with the same random seed 632 used in the Github code? Thank you.