llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

Published 20 hours ago • verified publisher icon ggml-org

[Review] Merge PowerInfer with llama.cpp mainline

Open chsasank opened this issue 1 year ago • 7 comments

Writing a review of PowerInfer with a view to merge into llama.cpp.

References:

  1. https://github.com/SJTU-IPADS/PowerInfer
  2. https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-20231219.pdf

Other discussions:

  1. https://news.ycombinator.com/item?id=38701822
  2. https://www.reddit.com/r/LocalLLaMA/comments/18luk10/wait_llama_and_falcon_are_also_moe/
  3. https://twitter.com/omarsar0/status/1737168751668187229

chsasank avatar Dec 20 '23 08:12 chsasank

Very nice. Just 26 commits. Of course there are some conflicts.

chsasank avatar Dec 20 '23 08:12 chsasank

That makes a lot of sense. Let me know what are the results that we need to verify from the paper before working on merging it. Pointers to specific sections/figures would help.

I think it'll be great if we can ensure llama.cpp remains mainline for inference research work.

chsasank avatar Dec 20 '23 12:12 chsasank

I've written about it here: https://github.com/ggerganov/llama.cpp/discussions/4534#discussioncomment-7900305

JohannesGaessler avatar Dec 20 '23 12:12 JohannesGaessler

Got it. Read through it. I'll let you guys discuss and let me know if it's worth working on this.

chsasank avatar Dec 20 '23 12:12 chsasank

MUST, HAVE, SPEEEEEEED

skidd-level-100 avatar Jan 02 '24 10:01 skidd-level-100

Any news on this ?

alkeryn avatar Apr 19 '24 15:04 alkeryn