how to search the hyper-parameter lamda0

[weihua04]

Thank u very much for open the sourcecode.

I have read the paper <Implicit Semantic Data Augmentation for Deep Networks>, and trained it in vehicle-classification task (with 20 id and 10thousand images)and vehicle-reid task (with 200 id and 10 thousand images) with our own dataset, and it worked respectively with resnest50 / resnet50 with lamda_0 = 7.5 . but when i change the backbone with resnet101 ,it didn't work with lamda=7.5,

1. Do u have any good idea to search the lamda_0?
2. Does it have relationship with backbone and task? 2.1) Does much ids in dataset can choose the lamda_0 larger? i found u choose lamda_0=0.5 on CIfar and 7.5 on imagenet , seems the more id with more difficult task can choose the lamda_0 large? (cause CIfar only has 10/100id ,while the imagenet id much larger) 2.2) How to choose lamda with deeper network? such as rensnet-101/152

[blackfeather-wang]

Thank you for your attention!

1. On ImageNet, we generally find that the optimal lamda_0 for ResNet is within {2.5, 5, 7.5}. Besides, similar to most regularization techniques, a relatively long training schedule might be helpful (e.g., 200 or 300 training epochs).

(2-1) We choose small lamda_0 on CIFAR as we use the entire covariance matrices. On ImageNet, we approximate the covariance matrices by their diagonals to save GPU memory, where we find larger lamda_0 may help(i.e., {1, 2.5, 5, 7.5, 10}). (2-2) We use 7.5/5 for ResNet-152 and ResNet-101, respectively. If this does not work, maybe you can try {1, 2.5, 10} to see how the performance changes.

If you have any other questions, please feel free to reach me.