distilabel
distilabel copied to clipboard
[FEATURE] Add Self-Taught Evaluator
Is your feature request related to a problem? Please describe.
Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve.
Describe the solution you'd like
integrate meta's self taught evaluator as a packaged step
Additional context
repo : https://github.com/facebookresearch/RAM/tree/main/projects/self_taught_evaluator paper : https://arxiv.org/abs/2408.02666