Have difficulties to reproduce transfer learning results in Table 8 and Table B.5

[Big-Brother-Pikachu]

Hi, thanks for your excellent work and for sharing the codes. I have some problems reproducing the results in Table 8 and Table B.5. I use the provided ImageNet-pretrained resnet50_1x model to do the linear evaluation of Cars and get poor results. Here are my settings:

optimizer: SGD lr:0.1 momentum:0.9 weight decay:0.0001 epochs:200 batchsize:256 We get the results 45.47%.

optimizer: L-BFGS lr:0.001 weight decay:0.0001 epochs:200 batchsize:256 We get the results 41.39%.

The results above are the best we can get using these two optimizers, but there are far behind 50.3% reported in the table. We follow the preprocess as written in 'As preprocessing, all images were resized to 224 pixels along the shorter side using bicubic resampling, after which we took a224×224 center crop.' We have tried linear layer w/wo bias, and whether freeze the batch norm statistics. So why do we still have a 5 points gap? Could you provide the hyper-params for ResNer 50 1x? Or suggest the most likely reason for this? Thanks a lot. (After looking up the issues, I find this issue, maybe @VSehwag and @simonster could help us?) I would appreciate it a lot if you can help us!:pray:

[VSehwag]

In my attempt to reproduce the results, I did a hyperparameter sweep for learning rate, momentum, and weight decay (2-3 values for each). Though the standard setting of (lr=0.1, wd=1e-4, m=0.9) works well in some cases, you will be surprised to see the gains by just bumping up the weight decay or momentum.

[Big-Brother-Pikachu]

Thanks a lot for your quick reply! In the last 2 hours, we improve our results to 47.08% (SGD lr=0.5, wd=0, m=0.9), as you suggest we should move on trying different hyperparameters setups. But I can't get good results using L-BFGS optimizer(about 41%) as suggested in the paper, do you have any idea about this problem? Thanks a lot again!

[deepankarvarma]

Can you share the notebook @Big-Brother-Pikachu , I have to replicate for the CIFAR-10

[Big-Brother-Pikachu]

@deepankarvarma, as it has been a long time, it is not easy to get back our codes for this. We have tried but only found some incomplete scripts and did not find the configs. We recommend looking for others' scripts as we find it hard for ourselves to comprehend the found codes. main_linear.txt