distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Add Self-Taught Evaluator

Open Josephrp opened this issue 5 months ago • 0 comments

Is your feature request related to a problem? Please describe.

Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve.

Describe the solution you'd like

integrate meta's self taught evaluator as a packaged step

Additional context

repo : https://github.com/facebookresearch/RAM/tree/main/projects/self_taught_evaluator paper : https://arxiv.org/abs/2408.02666

Josephrp avatar Sep 28 '24 08:09 Josephrp