How do you choose the optimal hyper-parameters?

[JinYang88]

There are several hyper parameters existing:

1. teacher model hyper parameters
2. student model hyper parameters
3. KD hyper parameters (e.g., balance weight for different losses)
4. Training hyper parameters (e.g., learning rate)

It is hard to enumerate for every combination, because it may explode. How do you find the best (or suboptimal) hyper parameter?

Thanks!

[surprisedong]

same question, just do not know how to set weight for different loss

[ShristiDasBiswas]

Hi, were you able to figure out a good set of hyperparameters?