zhangzai666

> 您好，请问您的预训练数据集哪里下载的

> > > > > > > > > 您好，请问您的预训练数据集哪里下载的 > > GLM的页面有一些数据的下载地址，比如wikitext的数据：https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip 感谢您的回答，我想继续预训练GLM-chinese。但是我发现代码中的数据集是.lazy文件。这个您这边有示例么

> 非常感谢您的回答，请问是这种类型的么： {"title":"XXX","content":""} {"title":"XXX","content":""} {"title":"XXX","content":""} 即一行一个json就可以么

> ### Is there an existing issue for this? > * [x] I have searched the existing issues > > ### Current Behavior > loss第一步有输出，接下来全部为nan > > ### Expected Behavior...

> > CUDA out of memory. Tried to allocate 6.28 GiB (GPU 1; 39.45 GiB total capacity; 31.41 GiB already allocated; 5.99 GiB free; 31.42 GiB reserved in total by...

> > CUDA out of memory. Tried to allocate 6.28 GiB (GPU 1; 39.45 GiB total capacity; 31.41 GiB already allocated; 5.99 GiB free; 31.42 GiB reserved in total by...

> > > > CUDA out of memory. Tried to allocate 6.28 GiB (GPU 1; 39.45 GiB total capacity; 31.41 GiB already allocated; 5.99 GiB free; 31.42 GiB reserved in...

> 已经修复，可以重新加载下 您好，感谢您的回复。 刚才试了加载chatyuanV2。您是加载词表吧extra_id的数量设置为0了，所以tokinzer的vocab_size减少了100.但是T5模型预训练期间需要extra_0到extra_100把。不应该是把模型的embdding层的维度增加为32228来适应extra_0到extra_100这100个mask词么

十分感谢，chatyuan无监督训练的数据集简单示例可以看一下么，用什么进行标记mask的

Ok, I really appreciate your answer