open_llama

open_llama copied to clipboard

tokenization issue for code

7

Does this still a bug for tokenization? I want to use this for code. Thanks!

brando90

brando90

I use FastChat as the framework for both training and dialog-based inference, and FastChat supports Meta/Llama. I was excited to try the 3B state Open-Llama model, and the FastChat finetuning...

RDouglasSharp

Code to train llama on tpus?

1

Where is the code to train llama on tpu's?

kathir-ks

ofirkris

My code ``` ef reinitialize_weights_gpt_neox_20B_inspired_4_llama2(model): """ Note: we nearly gpt-neox_20B (2022) & llama1 , llama2 (2019) does not say how they init I think gpt-neox_20B & llama2 both have pre-layernorm,...

brando90

I have a question regarding training stability. I downloaded the complete dataset of Redpajama v1 from Hugging Face and followed the parameter settings from the Llama1 paper for data mixture...

lyccol

I am quite new to this space, are there any resources anyone would recommend? Further, what I am looking to do is have a set of "Context" that I would...

coffeebe4code

Any plans to do openLlama2?

3

Use Llama2 model and train on all latest and more efficient (like SlimPajama vs redpajama) open datasets ? Just for the base model, then maybe open-assistant team can rlhf it

djaym7

How can you call yourself open anything when you can't even release your code?

AmanSinghal927