llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

Finetuning Models

Open ak2028 opened this issue 2 years ago • 2 comments

I followed the tutorial at train/finetune_example/mpt-7b-arc-easy--gpu.yaml and added an additional evaluation using icl_tasks: 'eval/yamls/tasks_light.yaml' in order to evaluate accuracy on ARC Easy. As the model finetuned, training loss decreased, but so did accuracy, which appears to be a bug.

I repeated this using the full ARC Easy training set and the same thing occurred. Is there a reason that finetuning causes training loss to decrease but accuracy on evaluation to decrease?

ak2028 avatar Aug 27 '23 17:08 ak2028

When you used all of ARC easy, can you share what changes you made to the YAML?

samhavens avatar Aug 29 '23 05:08 samhavens

Sure, I only changed: data_dir: train/finetune_example/arc-easy/ In arc-easy I have a train.jsonl

I downloaded the data from: https://huggingface.co/datasets/ai2_arc

ak2028 avatar Aug 29 '23 14:08 ak2028