Ethan He

add core_attention_bias_type to TransformerConfig

2

`core_attention_bias_type` is needed to use alibi from transformer engine https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/api/pytorch.html?highlight=alibi#transformer_engine.pytorch.DotProductAttention.forward

pretrained locator

1

Hi, Could you provide the model and code of the locator?

train.py OOM on TPUv3-8

9

``` 1 Successfully loaded and sharded model parameters! 2 0%| | 0/155 [00:00