tensor_parallel
tensor_parallel copied to clipboard
Does tensor_parallel support multi-node tensor parallel training?
I want to konw too.
@BlackSamorez Hope you can answer this question 😄😄
@BlackSamorez I have 2 servers with a total of 16 GPUs, so I would love to be able to use multi-nodes tensor-parallel to train a large language model, for example Bloom 176B. So I hope you can answer how to use multi-nodes tensor-parallel. Thank you very much
Is this solved? if so, how?
same question.
ahaha everybody have same problem but I think there is no feature like this but we absolutely need it. Recently I tried DeepSpeed which is developing from microsoft, maybe it has but Microsoft's code doesn't suppport Windows 😄