Is 8 x 4090 (24gb) gpu enough to reproduce your code?

[longmalongma]

Thank you for your work, I only have 8 x 4090 (24gb) gpu, is this resource enough to reproduce your code?

[boheumd]

I have not tried on the 24GB GPU before. But I think you can reduce the batch_size while increasing the accum_grad_iters and keeping the total_batch_size the same, so that less gpu memory is needed. For example, in this https://github.com/boheumd/MA-LMM/blob/main/run_scripts/breakfast/train.sh#L18

[longmalongma]

Ok, thanks for your reply! I see!