Distribute Computing?

Open zhangddjs opened this issue 9 months ago • 3 comments

I have multiple machines with many CPUs, but each machine's token is slow (2 t/s), is there any way to deploy BitNet distributely so that I can utilize all Idle CPUs to improve the token generation speed?

May 15 '25 03:05 zhangddjs

ref: https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/

May 15 '25 06:05 zhangddjs

what type of machine are you using? maybe you need to figure out why it is so slow, ideally it should be over 20 t/s on a recent released machine.

May 20 '25 02:05 sd983527

actually I am running it on a raspberry pi 4b, maybe 4 of this machine can together hold 20 t/s

May 20 '25 07:05 zhangddjs