August Moharrami

DPO Trainer Crashes on multi-gpu setup!

3

### System Info Kaggle Notebook With 2X T4 GPUs. [link to Kaggle notebook:] (https://www.kaggle.com/code/augustmurr/dpo-issue-recreationl) The issue does not occur when loading the model on one GPU (for example "cuda:0"), but...

# What does this PR do? Fixes #2241 Since the focus was on replicating the checkpoint merging methods from [the paper](https://arxiv.org/abs/2410.10801), I have covered only Linear, TIES, SLERP, and DARE-TIES...

# What does this PR do? Fixes #2112 just putting this up as a demo. I didn’t get feedback on whether this should be part of `trl` or `datasets`, so...