Janko comments

Repositories
Issues
Comments

Results 2 comments of


                                            Janko

[Bug]: Speculative decoding reports errors when loading target model using distributed inference (VLLM's offical Ray setup)

> Hi, I encounter the same error when trying to run [#12915](https://github.com/vllm-project/vllm/pull/12915) with DeepSeek-R1 model. > > My env: > > 1. vllm: built based on [[Model][Speculative Decoding] Add EAGLE-style...

[Model][Speculative Decoding] DeepSeek MTP spec decode

Hi, have you replicated the inference acceleration effect after enabling MTP on multiple nodes？ My envs: Ray cluster: two nodes of 8 x H20, ray status is right. It turns...