Rajkumar Ramamurthy

@boblee22 Probably by end of this month :)

We are a bit delayed on this. But it is coming soon. Is there a particular LM that you would like to train (that needs multi node setup) ?

We have a branch for this. https://github.com/allenai/RL4LMs/tree/add-accelerate-support But we are still testing it thoroughly before rolling it out.

- So each run will have some randomness due to the dataset creation (we randomly select val and test samples) due to the large size of the original dataset. -...

Agreed! This is on our to-do list too. If you are interested and have time, you can contribute..

@ghadiaravi13 I think this is probably because of the transformers library version that we adapted to hf_generation_utils.py. Once we upgrade it to recent versions, we can support this.

@mnoukhov Hey, will get back to you after checking the configs. There could be config errors which made NLPO unstable.

@rajammanabrolu Can you double-check the NLPO config?

Hey, there is no straightforward way to do this. Just adapt the policy implementation to pass these extra arguments.

Hey, you can turn off the model parallel by setting this flag https://github.com/allenai/RL4LMs/blob/aa5d337c4c587049e039d572042bf5c95926c3be/scripts/training/task_configs/synthetic_generate_increasing_numbers/blendorbot_ppo.yml#L41. This would wrap the model with DataParallel instead.