litgpt icon indicating copy to clipboard operation
litgpt copied to clipboard
verified publisher icon Lightning-AI

Use FSDP everywhere

Open carmocca opened this issue 2 years ago • 0 comments

Replaces DeepSpeed with FSDP

Requires https://github.com/Lightning-AI/lightning/pull/17845

Closes #116 Closes #177 Closes #169

Falcon 7b takes 32 GB max memory allocated using 2 devices and 32-true or bf16-mixed precision. Loss is NaN, requires further debugging into why.

carmocca avatar Jun 07 '23 01:06 carmocca