OLMo
OLMo copied to clipboard
Kebab7
This is the kebab config, a smaller version of the dirk config. Differences from dirk:
- untied weights
- weight decay on everything
- adjusted
mlp_hidden_sizeso we come out at 7B parameters including embeddings