A_Pipeline_Of_Pretraining_Bert_On_Google_TPU icon indicating copy to clipboard operation
A_Pipeline_Of_Pretraining_Bert_On_Google_TPU copied to clipboard

Published 20 hours ago • verified publisher icon pren1

VM and Disk size when traing Bert in google TPU

Open suchunxie opened this issue 2 years ago • 1 comments

Hello. Thank you very much for sharing your experience. I also want to pretrain a bert with google v3-8, and I want to ask about how to choose the VM Type and Disk size to ensure a bert training. I noticed that one example said that with 4GBs training corpus, in the create_pretraining_data step, there may cause about 400MB RAM. Is the Virtual Machine and Disk size need to bigger than the RAM maty take? (my corpus is about 1GB)

best wishes.

suchunxie avatar Apr 28 '22 05:04 suchunxie

I solved this problem by using TPU VMs. And after furth check, if using VM instance, just choose a fitable CPU or GPU , and change the Disk size will be okay.

suchunxie avatar Jun 14 '22 06:06 suchunxie