Dataset format restrictions

[401qingkong]

Can your engineering code receive data sets in json format, such as oscar-1G? It seems that only dataset provided with folder and subfolder can be received. The original text data of oscar is stored in JSONL format (one JSON object per line), and each JSON object contains an "id" field and a "text" field. The "id" field stores a sample number, and the "text" field stores a piece of text. I want to compare the llama2 performance between your project and megatron-lm. How can I use the same dataset?

[daviswer]

Hey sorry for late response, this came in just before winter holiday for us! We don't currently support JSON format, but we do support users implementing their own data formats. There's documentation on how to do so here - simply implement appropriate is_json, open, len, get, slice operations and the dataloader will consume data files in your desired format.