Restormer icon indicating copy to clipboard operation
Restormer copied to clipboard

Make q, k, and v contiguous to get better performance for normalize

Open CaoE opened this issue 2 years ago • 0 comments

Make q, k, and v contiguous to get better performance for normalize. After the rearrange operations for q, k, and v, normalizations on the last dim for q and k will be applied. The non-contiguous memory format makes the performance of normalize on the last dim poor.

CaoE avatar Aug 22 '22 06:08 CaoE