Arijit Nukala comments

Repositories
Issues
Comments

Results 2 comments of


                                            Arijit Nukala

[Feature]: Unblock LLM while handling long sequences / Handling multiple prefills at the same time

@schoennenbeck Not sure if this has been implemented or if this discussion is out of date but the following paper has a neat implementation that could solve this issue: https://github.com/hao-ai-lab/vllm-ltr

[Performance]: vllm Eagle performance is worse than expected

> After fixing these two issues, the acceptance length of vLLM is now very close to the official EAGLE code. Are these fixes implemented in a commit? Or did you...