albert icon indicating copy to clipboard operation
albert copied to clipboard

Published 20 hours ago • verified publisher icon google-research

pre-training ALBERT with fp16 and other optimizations

Open ahadsuleymanli opened this issue 4 years ago • 1 comments

Hi, how would one get around to pre-train ALBERT with fp16 weights? Also is it possible to train albert on multiple GPUs? Also it would be great if anyone used transfer learning for teaching the English ALBERT a different language and would share their experience with me.

ahadsuleymanli avatar Jan 15 '20 12:01 ahadsuleymanli

That experience would be very valuable. Also I think this paper is relevant in the topic: https://arxiv.org/abs/1910.11856

josecannete avatar Jan 16 '20 20:01 josecannete