Joe Cummings

> @kartikayk I'm having this same issue, but on the full fine tuned checkpoint. i can't go back and re-train the model with a new checkpointer (i used meta's checkpointer,...

> > 3- just eyeballing it, I'm not particularly sure about that, but It does seem so. there's a lot of repetition, the model hallucinates really bad even on english...

This is something we're working closely with the EleutherAI team on providing soon. For now, if you have enough RAM (and patience) you can try running on CPU - this...

> > A tokenizer pad_id of 0. We can update this if we really see a problem, but it covers all our current use cases. > > Is this true?...

Thanks for this quick fix @water-vapor ! Can you post the output of a run with this updated change for posterity?

I think we need to get new numbers for this in general, probably can do this through an automated process. Closing this for now.

We're definitely interested in adding more models! As I understand it, the 2B and 7B architectures are roughly the same (just different sizes for the parameters). If you'd be interested...

Hi @agutell, great questions! I think both of these use-cases are covered by this tutorial: https://pytorch.org/torchtune/main/tutorials/chat.html. If you have follow-up questions though, please let us know!

> Thank you! I'll check it out. One thing though, that page does not seem to show when you enter the documentation from [pytorch.org/torchtune/stable/index.html](https://pytorch.org/torchtune/stable/index.html) So we have a stable version...

> @joecummings should I add unit tests for this PR ? Whoops, I keep overwriting instead of quote and reply. Let's just start with W&B run first.