Restormer
Restormer copied to clipboard
Make q, k, and v contiguous to get better performance for normalize
Make q, k, and v contiguous to get better performance for normalize. After the rearrange operations for q, k, and v, normalizations on the last dim for q and k will be applied. The non-contiguous memory format makes the performance of normalize on the last dim poor.