Expected Data Format

[aflah02]

❓ The question

I was looking at the config files and noticed that the config files sometimes point to .npy files for the dataset. Is there any script to generate the same from a set of text files or any other format.

[aman-17]

You can use Hugging Face to download the dataset directly or the Dolma toolkit. The Hugging Face repository provides easy access to the dataset, and the Dolma toolkit offers utilities to handle different data format. If you need further help, feel free to follow up.

[baileykuehl]

Hi, thanks again for the inquiry! We’re currently working on closing out old tickets, so we’re closing this out for now, but if you require a follow-up response, please re-open and we will get back to you!

[aflah02]

Thanks Sorry I missed replying earlier