fsdp_qlora icon indicating copy to clipboard operation
fsdp_qlora copied to clipboard
verified publisher icon AnswerDotAI

Training using FSDP, qLoRa on multinode

Open alon-12345 opened this issue 10 months ago • 0 comments

How to train a TinyLlama-like model using FSDP, qLoRa on two (or more) nodes if each node has one (or more) GPU using train.py (main branch) ? I am grateful in advance for any help!

alon-12345 avatar Jan 29 '25 16:01 alon-12345