litgpt

litgpt copied to clipboard

In the finetuning scripts, we only allow

precision: Literal["bf16-true", "32-true"] = "bf16-true",

But we also use DeepSpeed when devices > 1. However, in this case, you'd get a

ValueError: `precision='bf16-true')` is not supported in DeepSpeed. `precision` must be one of: ('32-true', '16-mixed', 'bf16-mixed').

Should we allow bf16-mixed, or should we switch to FSDP? Or something else?

We can choose the precision based on whether deepspeed is used. I guess @awaelchli manually changed the precision value when trying out deepspeed in https://github.com/Lightning-AI/lit-llama/pull/128 (where this code originally comes from).

Note that FSDP also wouldn't work with bf16-true.