huggingface.js icon indicating copy to clipboard operation
huggingface.js copied to clipboard
verified publisher icon huggingface

Add Aphrodite Engine to Local Apps

Open AlpinDale opened this issue 1 year ago • 4 comments

This PR adds Aphrodite Engine to the list of local apps.

Aphrodite is a tensor-parallel LLM inference engine based on vLLM, with support for almost all transformers models and quantization formats. It currently supports:

  • Hugging Face Transformers
  • GGUF
  • ExLlamaV2
  • GPTQ
  • AWQ
  • Bitsandbytes
  • Smoothquant+
  • EETQ
  • AQLM
  • QuIP#

Deeplink support is not planned because it's a CLI-only app. This is my first time writing TypeScript, please let me know if I've made a mistake. Cheers!

Here's the SVG, if needed. pygchisel

AlpinDale avatar May 30 '24 04:05 AlpinDale

Thanks for the contribution @AlpinDale - massive fan of your Hub work too! Sorry for the delay on this!

cc: @julien-c

Vaibhavs10 avatar Jun 12 '24 19:06 Vaibhavs10

I noticed there was a PR for vLLM which streamlined the quantization stuff a lot better. I'll probably update this PR to follow that.

AlpinDale avatar Jun 12 '24 19:06 AlpinDale

ah i had missed that PR, thanks for pinging @Vaibhavs10!

aphrodite-engine looks cool 🔥

julien-c avatar Jun 13 '24 08:06 julien-c

Thanks for reminding me @Vaibhavs10 ! I'll work on this again tonight and hopefully we can finish it up.

AlpinDale avatar Aug 09 '24 10:08 AlpinDale