Woo-Yeon Lee comments

Results 5 comments of


                                            Woo-Yeon Lee

[Speculative decoding][Re-take] Enable TP>1 speculative decoding

Great! Thanks for the PR! I have much interest in vllm and this speculative decoding feature. Actually, I didn't know this was already in progress, and I was testing the...

[Speculative decoding][Re-take] Enable TP>1 speculative decoding

After a short experiment with this PR code, I'm getting almost the same result with my local branch, based on v0.4.2. I guess that it's due to the comm overhead...

[RFC]: Automate Speculative Decoding

@LiuXiaoxuanPKU Thank you (and other co-workers) for the great work! I checked the SmartSpec paper on arxiv and slides in the recent meetup, and it looks great :) I'm looking...

[Performance] [Speculative decoding]: Support draft model on different tensor-parallel size than target model

@cadedaniel Can I contribute my code that already implemented this feature on v0.4.2? I've referred to your code in #2188. I'm aware that #4933 is going on, so I want...

[Performance] [Speculative decoding]: Support draft model on different tensor-parallel size than target model

Thanks for the answer :) I'll send a PR maybe next week.