TinyLlama
TinyLlama copied to clipboard
jzhang38
The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Hello I was attempting to recreate this but with the tokenizer from llama3 (tiktoken) but with a few changes. I would be ok training a tiktoken from scratch if needed...
error: `ValueError: Could not load model TinyLlama-1.1B-Chat-v1.0 with any of the following classes: (, , ).` I have torch and accelerate libraries. Still getting this error.
```size mismatch for model.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([256, 2048]) from checkpoint, the shape in current model is torch.Size([2048, 2048]).``` there are error when I use ```model = AutoModelForCausalLM.from_pretrained(path)```...
