nicosouth

can i use this code to conduct self-supervised pre-training based on llama-7b? (Not the dialog format given in the example, My dataset is pieces of text) if i can, which...

Where are the beginning and end tokens of text added？

1

Where are the beginning and end tokens of text added？ I am looking for the start token and end token in the code where are they added. But i can...

Hello! I noticed that there are two special transformations in the previous introduction ('mix2s' and 'mix2t'). But I didn't find these two json files. Where should I download these two...

hi, I looked up a lot of information. but I still don't understand the difference between zero-3 and megatron with zero-2. they all split the model.

hello! i read your paper and github. i notice that you train the pre-trained model. but i don't know how much data did you use in the pre-train. can you...

[BUG]when map the dataset, i set the num_proc = 2 or 4, it will make mistakes.

8

Running tokenizer on dataset (num_proc=2): 0%| | 0/666 [00:00