Megatron-LM icon indicating copy to clipboard operation
Megatron-LM copied to clipboard

Speed up the creation of attention mask

Open yuantailing opened this issue 2 months ago • 1 comments

Prefer to use the inplace variant of triu_/tril_ because they are faster than the out-of-place variants since torch 2.3.0 (https://github.com/pytorch/pytorch/pull/115013).

yuantailing avatar Apr 29 '24 01:04 yuantailing

generally, mask will be created inside transformer engine if --use-mcore-models

ethanhe42 avatar May 02 '24 19:05 ethanhe42