Pedro Cuenca

> The instruct models need the `tokenizer.ggml.eos_token_id` to be 128009, or ``. @jxy Our comments were sent at the same time :) Yes, that's one of the solutions I mentioned,...

I'm a newcomer to the project so can't comment about past design decisions. Before #6144, I think `convert.py` was used to convert Llama/Mistral models (native weights **or** in HF transformers...

Regarding the conversion of the original tokenizer to pure BPE, the [transformers implementation is now available as a PR](https://github.com/huggingface/transformers/pull/30334/files#diff-08a7e5c7b50f73fc176e9a35899810080f0bc5b9e54278866f2b48ce68ddca30R1491). If anyone decides to tackle this, keep in mind the tiktoken...

> Any update on converting RAW meta models to HF?? You can use the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py) that was merged yesterday to transformers @ `main`.

Thanks a lot @ppham27!

> "Societal Impact" or "Safety" How about `Safety Assessment`, so the goal is to encourage the reporting of specific evaluations the model authors may have undertaken? "Societal Impact" might be...

Awesome, let me know so we can work on loading from the Hub too :)

Hi @tmabraham! Thanks a lot! And thanks for the ping, I missed your original PR. However, the diff is showing changes in all the cells for some reason. Would it...

Hi @ljaniszewski00! GPT2 is just a language model, and hasn't been trained to sustain chat conversations. It's trained to continue a text sequence with plausible text that may come after...

> If you use a diffusers LoRA I think this is already what will happen, the lora will download the mother model and use them. Yes, exactly! The mother model...