TensorRT-LLM icon indicating copy to clipboard operation
TensorRT-LLM copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Dynamic scaling not working on RoPe / rotary_scaling

Open TheCodeWrangler opened this issue 9 months ago • 5 comments 

          @byshiue can you try to see if dynamic scaling works? linear scaling works fine. if dynamic scaling doesnt work at all, then this is indeed a bug.

Originally posted by @avianion in https://github.com/NVIDIA/TensorRT-LLM/issues/1595#issuecomment-2112786968

Several users have experienced errors in running engine files which were compiled to use "dynamic" rotary_scaling

Is dynamic scaling supported at this time?

TheCodeWrangler avatar May 15 '24 16:05 TheCodeWrangler