Support for BLOOM

[LamOne1]

I'd like to request a support for BLOOM as it was pretrained on many languages

[carmocca]

This would require adding support for ALiBi. This is not a priority at the moment.

A related feature request is https://github.com/Lightning-AI/lit-gpt/issues/199

[Jeronymous]

+1

[rasbt]

Flash attention now supports Alibi btw, so in that case, Alibi support in Lit-GPT could be a bit easier.

[carmocca]

We don't support flash attention from flash-attn. Supporting alibi would warrant an entirely new model_alibi.py definition

[rasbt]

I think we should update this perhaps:

[carmocca]

We support flash attention via PyTorch's scaled_dot_product_attention, just not via Tri Dao's flash-attn. The former uses one of the latter's implementations internally.