GaLore icon indicating copy to clipboard operation
GaLore copied to clipboard

Published 20 hours ago • verified publisher icon jiaweizzhao

Memory issue

Open fakerybakery opened this issue 9 months ago • 0 comments

Hi, thanks for releasing GaLore! I'm running out of memory whenever I use a sequence length longer than 512, even if I use a smaller model. I can train a 7B model w/ a 512 sequence length on 24G VRAM, but I can't train a 5B model w/ a 8192 sequence length. Thanks!

fakerybakery avatar May 21 '24 18:05 fakerybakery