onnxruntime icon indicating copy to clipboard operation
onnxruntime copied to clipboard

Fix num splits bug

Open aciddelgado opened this issue 1 year ago • 1 comments

Description

Found a bug with num splits where the heuristic isn't being performed properly due to incorrect passing of sequence length to heuristic function.

Motivation and Context

We were experiencing significant performance issues with long sequence length with flash attention due to this misconfiguration.

aciddelgado avatar Aug 28 '24 18:08 aciddelgado

Please change all places of get_num_splits_and_buffer_sizes using total sequence length.

tianleiwu avatar Aug 28 '24 19:08 tianleiwu