RedPajama-Data Languages

Can you mention what languages covered in this dataset? based on the arXiv:2302.13971v1, LLaMA only covers this kind of languages : bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk. Is there possible to add some new low resources languages, like Indonesian for example. Thanks

Apr 19 '23 05:04 firqaaa

Same question.

Apr 19 '23 07:04 kugwzk

That's correct, we cover the same set of languages and these come from the wikipedia slice of the dataset. We will add support for more languages in the future (also low resource ones).

Apr 19 '23 10:04 mauriceweber

That's correct, we cover the same set of languages and these come from the wikipedia slice of the dataset. We will add support for more languages in the future (also low resource ones).

Does the dataset currently contain Chinese resources?

Apr 20 '23 03:04 hnnw

no, currently we only have the following languages in the dataset: bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk. We are planning to add support for more languages in the future.

Apr 20 '23 06:04 mauriceweber