Driss Guessous
Driss Guessous
Should this be input/set in the interface? for instance https://github.com/Dao-AILab/flash-attention/blob/43375aab2893018dfb7950db1cfa623c14946ad6/flash_attn/cute/interface.py#L276-L281
As a work around you can disable the cudnn backend via https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.enable_cudnn_sdp Would you mind opening an issue on PyTorch with a smallish repro, I can then forward to the...
> but comes at a massive performance cost. The performance should be the same as in 2.4.1 since this is the first release with cuDNN backend enabled. Can you link...
Now that: https://github.com/pytorch/pytorch/pull/144348 has landed, thanks @alexsamardzic we should have support for this. I am going to close this for now but if you run into any issues @vgoklani feel...
not particularly, I dont think pytest has broken that much internally since we have relatively vanilla usage, if someone did a full swap sure seems like BE project for a...
Closing as this is expected behavior for tinygemm
Do you have more context @merrymercy, I just ran sglang w/ ``` drisspg/stack/59 *1 ❯ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --torchao-config int4wo-128 \ --port 40000 --host 0.0.0.0 ``` And...
From the looks of it though, this will likely solve: https://github.com/pytorch/ao/pull/2277
cc @svekars do you know why the mermaid charts no longer w/ the new theme: https://docs-preview.pytorch.org/pytorch/ao/2361/torchao_vllm_integration.html#kernel-dispatch-bringing-external-kernels-to-vllm ?