flux icon indicating copy to clipboard operation
flux copied to clipboard
verified publisher icon bytedance

[QUESTION]some questions about allgather+gemm

Open ChrisRanger opened this issue 1 year ago • 1 comments

I'm puzzled by several questions.

  1. I noticed multi ring modes in allgather-gemm, like all_to_all ring pull or push. How does ring mode effect perfmance when PCI-e?

3.Why disabled local_copy in allgather-gemm crossnode?

4.Is there any difference on perfermance between using P2P and using NVSHMEM?

5.When chunk size is not same, can flux address this case?

thank you for your generous help.

ChrisRanger avatar Oct 09 '24 04:10 ChrisRanger

  1. for PCI-e machines, better use ring mode. all-to-all is for NVLink
  2. nop
  3. local_copy is not disabled?
  4. there should not be any difference. if you find a gap, please report.
  5. don't understand your question, can you make it more clear?

houqi avatar Oct 12 '24 08:10 houqi