Training ImageReward model on different budgets

[bhattg]

Hi! The paper mentions that the training for the ImageReward model is not easy and is sensitive to hyperparameters. In the section about hyperparameters, it says --" We find that fixing 70% of transformer layers with a learning rate of 1e-5 and batch size of 64 can reach up to the best preference accuracy."

Is this for the 8k budget? Can we get the suitable hyperparams for the other budgets?

Secondly, which part of the code freezes the transformer layers? Thanks!

[xujz18]

Thanks for your discussion! Firstly, the hyperparameters are for 8k budget (but the shuffle may differ, so it's worth trying a little bit different ones). Secondly, see https://github.com/THUDM/ImageReward/blob/main/train/src/ImageReward.py#L87-L99.

[bhattg]

Thank you very much!

[bhattg]

Hey will it be possible to provide the hyperparams for 1k and 4k settings as well? That will be very useful.

[xujz18]

The 8k hyper-parameters should only need smaller adjustments to accommodate 1k/2k/4k.

[bhattg]

Thanks! In your experience which hyperparameters were the most sensitive? I will try to tune them.