Why isn't SDPA supported in cuda within candle?

[Murad-Awad]

Hi all, forgive my ignorance but my understanding is that SDPA is cuda compatible but I can't find an implementation within candle_nn for it. I want to run some transformer models (mainly whisper) on T4 GPUs (which don't support flash-attention 2) and want to extract more performance. Optimally, would like something similar to the python transformers library where SDPA is by default used when able.

[Murad-Awad]

@LaurentMazare any thoughts here?

[EricLBuehler]

@Murad-Awad our SDPA impl is specialized for Metal currently, and only in the decode phase where there is no masking.

For CUDA, the equivalent would most likely be to use FlashAttention (the Candle ecosystem has V1, V2, and V3).

[Murad-Awad]

I see @EricLBuehler; this is related then (saw that you reacted to it too): https://github.com/huggingface/candle/issues/2726. Do you know why/if flash attention v1 would be supported within this repo? From what I can tell the flash attention module is for flash v2 only which is incompatible with T4s. I saw https://github.com/huggingface/candle-flash-attn-v1 but I'm confused as to how I can make use of it with candle transformers/kernels.

[Murad-Awad]

Bump on this @EricLBuehler.