TinyLlama icon indicating copy to clipboard operation
TinyLlama copied to clipboard

how to determine reasonable max steps?

Open ScottishFold007 opened this issue 1 year ago • 1 comments

Hello, you have a great program! It has been very beneficial! One question I would like to ask is how to determine reasonable max steps based on the amount of data available (e.g. tokens and model parameter count)?or do you have any good ideas in this regard?

ScottishFold007 avatar Jan 11 '24 08:01 ScottishFold007

Hi you can determine the max steps based on how much tokens you want to train when using cos lr schedule.

ChaosCodes avatar Feb 08 '24 14:02 ChaosCodes