[Chatllama] Training Japanese language for finetuning or training from scratch

[TakafumiYano]

To the chatLLaMA team, Thank you very much for this nice project. I looked at the model file and saw that the comment of compatiblity with training, so I thought it would be possible to train with a training dataset that we have prepared ourselves. If possible, can you give us some tips on how to train additional datasets? We would like to experiment and report back with documentation.

Thanks.

[diegofiori]

Hi @TakafumiYano, thank you for reaching out. Personally, I would suggest to mix your actual training data with some datasets available open-source as SHP and Anthropic HH RLHF, which are already supported by chatllama. If your focus is on not-english data I would suggest to start from a LLM that has already been trained (as a Language Model) on data sources containing the language you want. If you feel like you want to augment your own dataset, give a look to the methods we currently support for extending datasets and adapt them for generating new data in non-english language.

For using your own data, you have to format them in a compatible way with chatllama. See here for further info.

Feel free to share on this issue any problems or advancements you get 😄

[TakafumiYano]

Hi @diegofiori Thank you for explaining me details. We will try to make Japanese language model based on Bloom model. If we succeeded in training the modela and RLHF with this chatLLaMA. I will make the document and share this community.

Thank you.