[WIP] Train Rankgen ranking model for RLHF

[bth5032]

This PR is just a WIP to show the approach. As of now, I am training on the webgpt dataset, there are three main items I still need to accomplish before this is ready for full review.

1. Training on summary-from-feedback dataset
2. Add more test metrics.
3. Refactor out of notebook

That said, I'm happy to take feedback on the implementation or anything you think I might be overlooking. The only training run I did so far is here (still getting the hang of WandB, do we have a shared group for this?).

Thanks! #78

[theblackcat102]

@bth5032 your test loss seems pretty bad, any idea why its so bad? Here's a reference for other models link

You can also refer to my fork for consistency and training settings. I use huggingface Trainer and yaml for parameter settings tracking.

[bth5032]

@theblackcat102 Yeah, I noticed there was a lot of overfitting. In the past using these contrastive style losses I normally clip them, but looking at your code it doesn't seem like you're doing that. I do see you have an epsilon in there which I can try. Are you having better results? I'm also fine tuning the whole model, might be better off if I freeze the lower layers.

I'm not super familiar with the hugging face trainer, but I'll take a look. My general impression was that it's super useful if you want to work within the hf ecosystem for everything, but it's kind of a pain if you're doing something custom.

[bth5032]

Closing, superseded by https://github.com/LAION-AI/Open-Assistant/pull/313