OLMo icon indicating copy to clipboard operation
OLMo copied to clipboard

How many tokens were trained for 7B model.

Open mathfinder opened this issue 8 months ago • 1 comments

The paper and the readme both say that 2.5 T tokens were trained. However, the corresponding config says 2 T tokens. ReadMe: https://github.com/allenai/OLMo/blob/26392798cbc4d9ac3898bd2949e77042220bf3f8/README.md?plain=1#L49 Config:

https://github.com/allenai/OLMo/blob/26392798cbc4d9ac3898bd2949e77042220bf3f8/configs/official/OLMo-7B.yaml#L74C1-L74C13

mathfinder avatar Jun 10 '24 14:06 mathfinder