OLMo icon indicating copy to clipboard operation
OLMo copied to clipboard
verified publisher icon allenai

Try longer warmup (5k) at the 7B scale with mitch init, normal init and fan-in init

Open dirkgr opened this issue 2 years ago • 7 comments

This should be a very easy thing to try. Just one setting in the config.

dirkgr avatar Aug 18 '23 21:08 dirkgr

blocked by Jonathan for now, will start as soon as I get the greenlight on Mosaic cluster

ananyahjha93 avatar Aug 23 '23 21:08 ananyahjha93

  • longer warmup with normal init

ananyahjha93 avatar Aug 23 '23 23:08 ananyahjha93

started longer warmup + mitch init: https://wandb.ai/ai2-llm/olmo-medium/runs/p4zz4gid

ananyahjha93 avatar Aug 24 '23 00:08 ananyahjha93

Run seems to have crashed after 1300 steps: restarting from checkpoint

ananyahjha93 avatar Aug 24 '23 16:08 ananyahjha93

mitch init + longer warmup to 10k: https://wandb.ai/ai2-llm/olmo-medium/runs/mdzkub4i

ananyahjha93 avatar Aug 28 '23 20:08 ananyahjha93

normal init + long warmup: https://wandb.ai/ai2-llm/olmo-medium/runs/iuttxpt6

ananyahjha93 avatar Aug 29 '23 04:08 ananyahjha93

fan_in + long warmup: https://wandb.ai/ai2-llm/olmo-medium/runs/zrm8mbw8

ananyahjha93 avatar Sep 02 '23 23:09 ananyahjha93

Marking the items prior to Feb 29th as "closed".

dumitrac avatar Apr 30 '24 20:04 dumitrac