torchtune icon indicating copy to clipboard operation
torchtune copied to clipboard

Regression testing in torchtune

Open SalmanMohammadi opened this issue 6 months ago • 0 comments

Regression tests in our repo test our recipes with full-size models, and run on a nightly basis. We currently only have a single regression test. This test finetunes Llama2-7B with LoRA for 10 steps then runs our Eleuther eval recipe to check the accuracy of the finetuned model on the truthfulqa_mc2 task #1305.

We should expand coverage of our regression tests across our recipes, and aim to also include tests for memory or performance for some of our models.

SalmanMohammadi avatar Aug 11 '24 14:08 SalmanMohammadi