LESS icon indicating copy to clipboard operation
LESS copied to clipboard
verified publisher icon princeton-nlp

Using Multiple GPUs

Open SD325 opened this issue 1 year ago • 3 comments

How can we utilize multiple GPUs for the gradient feature collection step? The current implementation only works with a single GPU.

SD325 avatar Mar 12 '24 19:03 SD325

Hi, currently gradient computation with multiple GPUs is not supported mostly because I have not found a way to use vmap to calculate gradients for each instance within a batch. If you'd like to use multiple GPUs, you can split your data into multiple files and launch a separate job for each subset of data.

xiamengzhou avatar Mar 15 '24 17:03 xiamengzhou

Hi, If I launch separate jobs, will the projection matrix remain the same when I set a same number for model_id?

simplelifetime avatar May 16 '24 03:05 simplelifetime

Yes! It will

xiamengzhou avatar May 16 '24 20:05 xiamengzhou