Does it support lora and pipeline parallel now?

[blldd]

I found that the memory usage is very large even when using zero3 and lora, so I was wondering whether can I support pipeline parallelism or Tensor parallelism?

[yaozhewei]

LoRA is supported. Pipeline parallelism and Tensor parallelism is not supported yet.

[taishiciR]

@yaozhewei hi~,请教一下。“finetune with Pipeline parallelism and Tensor parallelism” --后续官方有支持计划吗？我理解如果模型参数非常大情况下（e.g :bloom 176b），需要跨节点分布，此时需要pp支持吧，还是说zero策略目前原理上可支持参数和激活的跨节点？

[yaozhewei]

We tested up to OPT-175B and the current framework works well. We will discuss internally about the support of PP and TP :)

[yaozhewei]

Close the issue since there is no followup. Please reopen it if necessary

[LSX-Sneakerprogrammer]

Hi, do you have any plans on supporting Pipeline parallelism now?

[FlyCarrot]

We tested up to OPT-175B and the current framework works well. We will discuss internally about the support of PP and TP :)

Hi, can you give more infos about your machine's number and training efficiency? Thanks!

I am trying load 7B actor model and 7B rw model in Zero-3, if I use more than One Machine, it will very slow.(latency 120s -> lantency 1700s)