Rainy

Results 2 issues of Rainy

after this pr https://github.com/sgl-project/SpecForge/pull/290 fixed VLM's lack of support for set_aux_hidden_states_layers, but after this pr https://github.com/sgl-project/SpecForge/pull/308 This part of the code has been removed, which will cause bugs in VLM.

Hi, thanks for your great work on SGLang and SpecForge! I am trying to test https://huggingface.co/Rayzl/qwen2.5-vl-7b-eagle3-sgl on Qwen2.5-VL using the reference configs from: https://github.com/sgl-project/SpecForge/pull/102 , but the speculative decoding performance...

bug
low accuracy