Any descriptions on the dataset for pre-training?

[zhoudaquan]

Hi authors,

Congratulations on your great work! I have read through the paper. I found that there is no description on the source of dataset used for pre-training. Can you please share some information on which dataset or how you collect the dataset for pretraining?

Regards, DQ

[Maxlinn]

also interested in the dataset