is_causal=attention_bias is None

[nkkbr]

🐛 Describe the bug

https://github.com/allenai/OLMo/blob/5789cfe32390a0e80417e98285647cb8b41029ae/olmo/model.py#L598-L605

should the line 604 be is_causal=attention_bias is not None ?

Versions

Python 3.10.14

[epwalsh]

Hey @nkkbr, when attention_bias is not None, the attention_bias includes a causal mask, so we don't to set is_causal to True since that would be redundant.

[baileykuehl]

Hi, thanks again for the inquiry! We’re currently working on closing out old tickets, so we’re closing this out for now, but if you require a follow-up response, please re-open and we will get back to you!