Vincent Min comments

Results 3 comments of


                                            Vincent Min

Whenever I use QLoRA to train LLama/LLama 2 on an instruction-tuning dataset like Dolly or Alpaca I get a periodically oscillating training loss

@BugReporterZ Could you explain the reasoning for why group_by_length may be causing this issue?

Whenever I use QLoRA to train LLama/LLama 2 on an instruction-tuning dataset like Dolly or Alpaca I get a periodically oscillating training loss

@ritabratamaiti Yes, QLoRA is supported by SFTTrainer. You can use this example script and set `load_in_4bit=True` and `use_peft=True`. https://github.com/lvwerra/trl/blob/main/examples/scripts/sft_trainer.py See this blog for more details: https://huggingface.co/blog/4bit-transformers-bitsandbytes @BugReporterZ thanks for the...

the output only hava adapter_config.json,adapter_model.bin

Hi @pacman100 , could you explain why the code is structured such that you must provide the `base_model`? It seems to me that the `base_model` is already present in the...