How to Add PEFT to PPO Trainer or PPO Config

[ZhichaoWang970201]

I am trying to realize RLHF through PPO.

May I ask how can I realize PEFT in RLHF/PPO. I can see this parameter in DPOTrainer. However, I cannot see that in PPOTrainer.

[qgallouedec]

PPO does not yet support PEFT. But it may be a good enhancement.

[ccs96307]

Hi @qgallouedec, I'd love to contribute to this enhancement! I noticed that this issue has been open for three months, and I'd like to help bring it forward.

Is there anything specific I should be mindful of as I start on this? Any guidance or notes on the current status would be greatly appreciated.

Thank you!