Megatron-LM icon indicating copy to clipboard operation
Megatron-LM copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

add core_attention_bias_type to TransformerConfig

Open ethanhe42 opened this issue 5 months ago • 2 comments

core_attention_bias_type is needed to use alibi from transformer engine https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/api/pytorch.html?highlight=alibi#transformer_engine.pytorch.DotProductAttention.forward

ethanhe42 avatar Jan 21 '24 10:01 ethanhe42