[chat] generate process is not a single step in RL

[ht-zhou]

Hi I am from the ColossalAI team. I found that there are similarities between DeepSpeedChat and ColossalChat. We found that there might be some implementation error in our code, thus possibly leading to some convergence problems. If you refer to our implementation, you may encounter similar problems as well. We are planning to fix the bug these two weeks.

Meanwhile, it would be appreciated if you could reference our work if you adapted from our implementation.

[piekey1994]

The implementation of their ppo seems to be more similar to trlx. I use your colossalai ppo without gae algorithm, and the model can continuously optimize 20 generate epochs, but with trlx implementation, it may only optimize about 5 epochs. Can you share the implementation errors in colossalai that you mentioned?

[Randolph-zeng]

For the most libs I believe they have some extent of similarity(e.g. TRL, TRLX, ClossalAI, RL4LM etc), so I am not sure if they are referencing your code or more generally the instructGPT methodologies ... Though it will be much appreciated if you can elaborate more on the bugs you encounter