open_clip icon indicating copy to clipboard operation
open_clip copied to clipboard
verified publisher icon mlfoundations

How to estimate the memory usage of a gpu under the global loss setting?

Open Adenialzz opened this issue 2 years ago • 0 comments

I have noticed that under the global loss setting, when training with multiple nodes, the GPU memory usage will be much higher than when training with a single node. So, how should we estimate the GPU memory usage with multiple nodes in this situation?

Adenialzz avatar Jul 20 '23 11:07 Adenialzz