Usman Zia comments

Repositories
Issues
Comments

Results 3 comments of


                                            Usman Zia

Whenever I use QLoRA to train LLama/LLama 2 on an instruction-tuning dataset like Dolly or Alpaca I get a periodically oscillating training loss

@BugReporterZ What is the impact on training if we disable group_by_lenght? Is it comparable to it being set to true and only gain is saving memory?

Whenever I use QLoRA to train LLama/LLama 2 on an instruction-tuning dataset like Dolly or Alpaca I get a periodically oscillating training loss

Got it, thank you

[Feature request] Add custom dataset compatibility

HI @dred0n Just to confirm, I have formatted my dataset complaint to alpaca format i.e Instructions, input and output, i would just need to specify "dataset_format" as alpaca and name...