mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Need parallel linears

Open EricLBuehler opened this issue 10 months ago • 4 comments

  • [ ] RowParallelLinear
  • [ ] MergedColumnParallelLinear
  • [ ] QKVParallelLinear

EricLBuehler avatar Apr 01 '24 12:04 EricLBuehler

Is the plan to implement tensor sharding for both quantized and non-quantized versions?

hugoabonizio avatar Apr 03 '24 00:04 hugoabonizio

Yes, that is the plan.

EricLBuehler avatar Apr 03 '24 11:04 EricLBuehler

I'm quite interested in the parallelization in CUDA of the quantized models!

hugoabonizio avatar Apr 03 '24 12:04 hugoabonizio

Yes! We are beginning work on this topic now.

EricLBuehler avatar Apr 04 '24 14:04 EricLBuehler