DeepSpeedExamples icon indicating copy to clipboard operation
DeepSpeedExamples copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Why is the PPL so high in the beginning of Step-1 (SFT)?

Open siddharth9820 opened this issue 8 months ago • 0 comments

https://github.com/microsoft/DeepSpeedExamples/blob/737c6740bec38b77a24a59135b6481a53d566b38/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/training_log_output/opt-1.3b-globalBatchSize128.log#L4

Why is the PPL here 4k when we are starting with a pretrained model?

siddharth9820 avatar Oct 28 '23 03:10 siddharth9820