verl
verl copied to clipboard
volcengine
Support Generative Reward Model (GenRM)
According to the documentation, veRL only supports AutoModelForSequenceClassification. What would be the best way to implement generative reward model (GenRM) for veRL? I tried looking at FSDP Workers and Megatron-LM Workers but they no longer exist.
Instead of using a linear predictor, GenRM leverages CoT and next-token prediction to provide reward. GenRM is proven to be more accurate. https://arxiv.org/abs/2410.12832
Instead of using a linear predictor, GenRM leverages CoT and next-token prediction to provide reward. GenRM is proven to be more accurate. https://arxiv.org/abs/2410.12832
Are there any pretrained GenRM that we can play around with?
Instead of using a linear predictor, GenRM leverages CoT and next-token prediction to provide reward. GenRM is proven to be more accurate. https://arxiv.org/abs/2410.12832
Are there any pretrained GenRM that we can play around with?
I’m not aware of any existing pretrained GenRM. However, a basic GenRM could be created by simply prompting an LLM to act as a judge. For example, a prompt like this could be used:
Please act as a judge and provide a score from 1 to 5, with 5 being the highest quality. Think step by step before deciding on a score, and output the final score in \box{}.
I’m happy to help implement this feature, but I could use some guidance on the best place to integrate this reward worker.
Instead of using a linear predictor, GenRM leverages CoT and next-token prediction to provide reward. GenRM is proven to be more accurate. https://arxiv.org/abs/2410.12832
Are there any pretrained GenRM that we can play around with?
I’m not aware of any existing pretrained GenRM. However, a basic GenRM could be created by simply prompting an LLM to act as a judge. For example, a prompt like this could be used:
Please act as a judge and provide a score from 1 to 5, with 5 being the highest quality. Think step by step before deciding on a score, and output the final score in \box{}.I’m happy to help implement this feature, but I could use some guidance on the best place to integrate this reward worker.
Hi, would you please introduce some best practices of using this kind of gen rewards? That will be of great thank to you.
here's a naive implement of LLM as a Judge which runs as a new role in RL training. we use qwen model as a verifier, it could be replaced with any pretrained GRM as you like https://github.com/volcengine/verl/pull/1953
