Zikun Wu

How to OPT model with PowerInfer?

7

# Prerequisites Before submitting your question, please ensure the following: - [x] I am running the latest version of PowerInfer. Development is rapid, and as of now, there are no...

# Prerequisites Before submitting your question, please ensure the following: - [x] I am running the latest version of PowerInfer. Development is rapid, and as of now, there are no...

[Feature Request]How to measure the generation throughput(token/s)?

9

### Prerequisites - [x] I have searched existing issues and reviewed documentation. ### Problem Description I want to measure the DeepSeek-v2-Lite-Chat throughput of MoE-infinity using RTX 4080 Super(16GB).The code I...

Does it support other DeepSeek models?

4

I want to inference other DeepSeek models in V100 GPU.Does it support?Such as deepseek-ai's DeepSeek-R1-Distill-Llama-70B or DeepSeek-R1-Distill-Qwen-32B?