Unable to replicated training results

[ShechemKS]

We have been attempting to train the TRN model on the THUMOS'15 dataset, and have used resnet and inception to extract the features from the RGB and optical flow images respectively. The optical flow was computed using Farneback algorithm on OpenCV.

We were not training using the full dataset, only a subset of 120 videos, with a batch size of 8. However, we are unable to get any reasonable results, with the mAP score for any given class not exceeding 8%, after over 60 epochs. Would you have any insights as to why this is?

Additional note:- We also tried extracting features using different models, but attained the same results.