How to use lora to train the 30b model on multiple machines and multiple cards?

[Awyshw]

[ZYHowell]

We've tested that the current script is runnable on multiple cards, a single machine(8 x 40GB A100) using DeepSpeed ZeRO-3. The multi-node case is not tested yet. It's WIP

[cason0126]

We've tested that the current script is runnable on multiple cards, a single machine(8 x 40GB A100) using DeepSpeed ZeRO-3. The multi-node case is not tested yet. It's WIP

still Doing？

[gebilaoman]

mark

[surak]

@ZYHowell any progress on this? There was an issue about using slurm I've seen around as well...