Do evaluation during training

[Daniel-1997]

When I read the parameters to be set to finetune.py, I am a little confused. since there are several parameters about evaluation during training:

-- validation_file: I did not find the tutorial about how to make a validation text file. -- do_eval: I think the "dev set" refers to the validation set, am I right? -- evaluation_strategy: I wonder if I set it to "steps", it will evaluate after per step training? -- eval_steps: Will it work when do_eval was set to False? I tried to set this parameter, however, I met an error. -- validation_split_percentage : I tried to set this parameter, however, I met an error.

The error information says that I need to set validation set if I want to do evaluation, but I have set validation_split_percentage and eval_steps. After I read the code , I found that when building the transformer trainer, the eval_dataset is set to None, so I guess this is the reason?

if I want to do evaluation during training , what should I do ? look forward for your reply, thank you!

[research4pan]

Thanks for your interest in LMFlow! We currently disable the old evaluation logics and will add a new evaluation logics in the future. The old evaluation process will only output the loss in a splitted validation set, which is not powerful enough to satisfy our future demands.

As a result, those arguments are now temporarily deprecated. You may remove --do_eval and --validation_file for now. We will let you know once the new evaluation procedure is added. Thanks for your understanding 🙏

[shizhediao]

This issue has been marked as stale because it has not had recent activity. If you think this still needs to be addressed please feel free to reopen this issue. Thanks