Tri Dao

icon indicating a website link https://tridao.me

icon location Princeton, NJ icon location Assistant Professor @ Princeton CS, machine learning & systems

Results 473 comments of Tri Dao

Softmax (particularly exp operations) becomes a major bottleneck in full FP16 pipeline

Yeah overlapping works reasonably well for Hopper, for older architectures it might be harder to do.

Softmax (particularly exp operations) becomes a major bottleneck in full FP16 pipeline

I don't think that's easy to measure. For small hdim (e.g. 64) and for fp8, softmax is still a bottleneck.

Avoid padding computation with `cu_seqlens`

Yes I think that should work. You should test that still