trlx
trlx copied to clipboard
[WIP] Add Minimum Risk Trainer support
Initial working version of MRT.
Besides the difference in the learning procedure, most of the MRT types are pretty similar to the PPO ones, such as MRTRLElement
. The main difference is that MRT generates a batch of candidate sentences for the same prompt instead of a single sentence per prompt.
Main changes:
- Add AccelerateMRTTrainer
- Introducing MRT configs/data types (
trlx/data/mrt_types.py
, - Example for translation:
examples/mrt_translation_t5.py
I've currently tested it with t5 on translation and summarization tasks.
TODOs:
- [ ] Test on decoder-only models
- [ ] Make sure it is working for
- [ ] Add support for
MarianMT
models - [ ]
debug_mrt_summarize_daily_cnn_t5.py
(only for debugging)
Please let me know if there are any other suggestions as well. cc @LouisCastricato