can it make Lora sft？

[ReverseSystem001]

Limited by graphics card devices. For most people, Lora is the only way to fine-tuning. can it make lora sft?

[CoinCheung]

Hi, thanks for paying attention to this !!

This repo is currently designed for full parameter finetuning, but lora freezes most of the parameters. Since they contradict with each other, currently this repo does not support lora.

This repo bases on pipeline method, which allows you to train your model with DP + PP (megatronLM is DP + PP + TP, the so-called 3D layout). This is faster and requires less memory than zero-based methods when there is not so many gpus (100+). You can train a 7b or 13b model on a server with 8 gpus (24G), which I believe many companies can afford to.

[ReverseSystem001]

great job

---Original--- From: @.> Date: Tue, Nov 7, 2023 09:36 AM To: @.>; Cc: @.@.>; Subject: Re: [CoinCheung/gdGPT] can it make Lora sft？ (Issue #24)

Hi, thanks for paying attention to this !!

This repo is currently designed for full parameter finetuning, but lora freezes most of the parameters. Since they contradict with each other, currently this repo does not support lora.

This repo bases on pipeline method, which allows you to train your model with DP + PP (megatronLM is DP + PP + TP, the so-called 3D layout). This is faster and requires less memory than zero-based methods when there is not so many gpus (100+). You can train a 7b or 13b model on a server with 8 gpus (24G), which I believe many companies can afford to.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>