LMFlow icon indicating copy to clipboard operation
LMFlow copied to clipboard

[Feature] LISA multi GPU support

Open wheresmyhair opened this issue 5 months ago • 0 comments

Description

LISA now supports multi GPU. Key points:

  1. When initialize models, ds will wrap all model params with optimizer, which uses huge amount of gpu mem. However, according to LISA's logic, we do not need to wrap all params with optim, since only activated layers will be updated. So we hack the initialization such that trainer only wrap one layer at the beginning (to avoid CUDA oom error).
  2. Currently cannot save training args when using LISATrainer, as it will raise error when pickling. Model weights can be saved correctly.

wheresmyhair avatar Sep 25 '24 04:09 wheresmyhair