[Question] Support Pipeline Parallelism

[SeungsuBaek]

Hi.

I want to use pipeline parallelism and tensor parallelism or combination of it.

In the examples directory, only internLM supports two parallelism and other models support just one of parallelisms.

Is there any solution to use pipeline and tensor parallel together with GPT or OPT models?

and why pipeline parallelism does not implement well?

Thanks for reading.

---

2024.02.03

internLM also does not support pipeline parallel.

examples/internlm/convert_checkpoint.py line 1558