Flash Attention 3 fp8 support 4090?

[huanpengchu]

trafficstars

My device is a 4090 in hopper architecture, consistent with the h100 architecture. But on the homepage it says “Requirements: H100 / H800 GPU, CUDA >= 12.3.”

I would like to know if flash attention supports the 4090.

[tridao]

4090 is Ada, not Hopper

[han508]

@tridao I wonder if it's possible to support 5090?

[hg0428]

@han508 5090 is BlackWell. I need Flash Attention 3 FP4 BlackWell support and soon! Or I'll have to write the code myself which will be time-consuming. I'm starting the training runs in April.

[mnicely]

@hg0428 can you offer insight into why you need FA FP4 support? Have you tested the accuracy?

[hg0428]

@hg0428 can you offer insight into why you need FA FP4 support? Have you tested the accuracy?

Give this a good read: https://arxiv.org/pdf/2501.17116