Has the data been shuffled?

[Lisennlp]

Hello, I see your batch_view.py, found that the data does not use a shuffle, but in the gpt-neox library, the data is shuffled. So I want to make sure that the author did or did not shuffle during the training? Hope to get your answer, thank you！

[pietrolesci]

I think this might provide an answer https://github.com/EleutherAI/pythia/issues/123#issuecomment-1878882214

[itsnamgyu]

The data is shuffled in terms of documents. The repo-id says preshuffled in https://github.com/EleutherAI/pythia#exploring-the-dataset, i.e., EleutherAI/pile-standard-pythia-preshuffled.

I'm actually not sure about https://huggingface.co/datasets/EleutherAI/pythia_deduped_pile_idxmaps mentioned in https://github.com/EleutherAI/pythia#reproducing-training. I will add a quesiton about this on #123.