Why do we have 2 scripts for fine-tuning?

[samin-batra]

trafficstars

Hello,

I have been experimenting with fine-tuning StarCoder and I see there are 2 different scripts for fine-tuning, both of which handle the data processing differently and also, one uses deepspeed while the other doesn't. One is using LORA with PEFT while the other doesn't and thus keeps giving OOM when run on a single A100 80GB GPU. Also, have noticed that while running the script under finetune directory, it doesn't log anything. Is it something to do with how the logging steps parameter and the max steps are related to each other? Has anyone else faced this issue?

Thank You

[ArmelRandy]

Hi. The issue you have about not having any log is certainly related to your logging_steps, you should decrease --log_freq. The 2 scripts are different and thus have different hardware requirements. The script is chat was used to fine-tune StarCoder on 8 A100 (80G) as you can read here.

[samin-batra]

Hello,

Thank you for your reply. I have decreased the log_freq as well to log for every step but still the trainer.train() call gets stuck. It only outputs the number of examples, the number of epochs (which is a huge number), and other information related to hyperparameters provided. It doesn't print any logs related to loss and epochs which you'd normally expect to appear. Also, regarding the chat directory, can we run a fine-tuning on just 1 GPU (A100 80GB)?

[ArmelRandy]

You should be able to see some logs, you can try --log_freq 1 to be sure a wait a bit longer. I have not tried to a full fine-tuning of StarCoder with that script on less that 8 A100 so I can not confidently respond to your question. You can try it with all sort of tricks (short sequence length, batch_size = 1, gradient checkpointing etc.).