Cody Yu
Cody Yu
今天為了打出 "市長" 將 "是" 修正為 "市"，然後就無法修正回來了。 現在所有的 "市" 第一候選字都是 "市"。
Hey I got the same error with PyTorch 2.3.0 and flash-attn==2.5.8. Is there anything changed in the wheel? ``` flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi ```
This is on my TODO list for a while but I haven't got a chance to work on it. If anyone wants to contribute a PR before I do, I'll...
Thanks for the contribution and this looks exciting! Some thoughts/questions: 1. Have you benchmarked on other GPUs, such as H100, A10g and L4? 2. Why it only works for flash...
> I suggest keeping them separate to allow us to (1) use different kernels if required in future based on model or use-case and (2) to continue to have support...
Thanks for the improvements! Do you have some end-to-end benchmark results to share with?
Thanks for the numbers. In this case it would be great to come up with tuned configurations (we have them for some NVIDIA GPUs under https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/fused_moe/configs). Also how does this...
Awesome! Do you want to add AMD tuned configs in this PR or you prefer to have a follow-up PR for it?
Ok I'll ping folks to force merge.
FYI: @sroy745 has #8678 verifying the functional correctness. Could you folks coordinate on this?