trlx icon indicating copy to clipboard operation
trlx copied to clipboard

[WIP] Add Minimum Risk Trainer support

Open alexandremuzio opened this issue 1 year ago • 0 comments

Initial working version of MRT. Besides the difference in the learning procedure, most of the MRT types are pretty similar to the PPO ones, such as MRTRLElement. The main difference is that MRT generates a batch of candidate sentences for the same prompt instead of a single sentence per prompt.

Main changes:

  • Add AccelerateMRTTrainer
  • Introducing MRT configs/data types (trlx/data/mrt_types.py,
  • Example for translation: examples/mrt_translation_t5.py

I've currently tested it with t5 on translation and summarization tasks.

TODOs:

  • [ ] Test on decoder-only models
  • [ ] Make sure it is working for
  • [ ] Add support for MarianMT models
  • [ ] debug_mrt_summarize_daily_cnn_t5.py (only for debugging)

Please let me know if there are any other suggestions as well. cc @LouisCastricato

alexandremuzio avatar Apr 10 '23 05:04 alexandremuzio