Support for mutliturn online RL training

[UbeCc]

Currently, verl only support single-turn rl training. As agents turning is becoming urgent, will verl support multiturn rl in the next few days? Maybe I can help. Thanks!

@PeterSH6 @zhaochenyang20

[zhaochenyang20]

Sure. Welcome to join. And, proposal should be professional, look at how I make proposal to SGLang:

Reaserch Project 开题

1. 问题是什么，如何清晰定义问题；
2. 这个问题的 scope 如何，有什么非常强的假设，这些假设合理么；
3. 谁在乎这个问题，不要说谷歌会在乎这个问题，要精确到谷歌的某个组甚至是某个人会在乎这个问题；
4. 现有的方案是什么，有什么不足；
5. 我们的方案可能是什么（开题的时候不一定能写完）；
6. 如何评估？达到了什么效果则说明我们的方法奏效了；
7. 这个过程有什么不确定性，风险如何；
8. 计划工作的 timeline 是如何；

Feature 开题

1. 目标框架现在的实现是什么样子的；有什么问题；
2. 修改方案是什么，预计会修改什么部分；
3. 将会达到的效果如何；
4. 有什么不确定性；
5. 计划工作的 timeline；

[PeterSH6]

@UbeCc Nice suggestion! We can discuss the plan this week. Could you connect with us through WeChat or Slack?

[zhaochenyang20]

@UbeCc @PeterSH6 I can connect you guys. Haoran is my senior. And, good night 😂

[UbeCc]

@UbeCc @PeterSH6 I can connect you guys. Haoran is my senior. And, good night 😂

Thanks Chenyang, enjoy your day!

[UbeCc]

@UbeCc Nice suggestion! We can discuss the plan this week. Could you connect with us through WeChat or Slack?

Yeah, let me send my WeChat id through email

[YSLIU627]

Great idea! I could also offer some help!

[AIBionics]

I'm also working on multiturn online RL training at the moment, and I'd be glad to assist if you need any help. Maybe we can create a WeChat group and then add everyone to the group for discussion.

[UbeCc]

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

[email protected]

[YSLIU627]

Great! My wechat is liuzhihan0627 . See you then, Best, Zhihan

On Tue, Feb 25, 2025 at 6:45 PM Haoran Wang @.***> wrote:

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

@.***

— Reply to this email directly, view it on GitHub https://github.com/volcengine/verl/issues/385#issuecomment-2683611993, or unsubscribe https://github.com/notifications/unsubscribe-auth/APK6TFYXJJBZODWCCJZJ3RL2RUFB7AVCNFSM6AAAAABX3D4UF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGYYTCOJZGM . You are receiving this because you commented.Message ID: @.***> [image: UbeCc]UbeCc left a comment (volcengine/verl#385) https://github.com/volcengine/verl/issues/385#issuecomment-2683611993

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

@.***

— Reply to this email directly, view it on GitHub https://github.com/volcengine/verl/issues/385#issuecomment-2683611993, or unsubscribe https://github.com/notifications/unsubscribe-auth/APK6TFYXJJBZODWCCJZJ3RL2RUFB7AVCNFSM6AAAAABX3D4UF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGYYTCOJZGM . You are receiving this because you commented.Message ID: @.***>

[sbl1996]

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

[email protected]

I've already wrote a multi-turn implementation and it works, but the training is not stable. Could you add me to the wechat group? My wechat id is Zukala-Koth. Great thanks! @UbeCc

[zhaochenyang20]

@sbl1996 Sure. I will tell him tommorow.

[UbeCc]

Yeah could you plz send your WeChat id to me through email? Let me create a group and work together. [email protected]

I've already wrote a multi-turn implementation and it works, but the training is not stable. Could you add me to the wechat group? My wechat id is Zukala-Koth. Great thanks! @UbeCc

Done. Thank you!

[oliverz20]

Hi, I’m interested in multiturn RL as well. Could you please add me to the group? My WeChat ID is sfoliver. Thanks a lot! @UbeCc

[Tshiyao]

Hi @UbeCc , I'm also really into multiturn RL and would love to join the group! My WeChat ID is Liu_Qihuang . Looking forward to connecting and learning more. Thanks!

[UbeCc]

Got it. We are already working currently. Thanks for your support

[Jackory]

Hi @UbeCc , I'm also interested into multiturn RL and would love to join the group! My WeChat ID is innerpeace . Looking forward to connecting and learning more. Thanks a lot!

[hongyi-zhang]

@PeterSH6 @UbeCc @zhaochenyang20 I'm interested in multi-turn RL as well. We have a real-world use case and I was going to start my own implementation before seeing this thread. Would love to contribute or discuss technical design, whichever is preferable!

[LeslieTrue]

@UbeCc @PeterSH6 @zhaochenyang20 Interested in contribution! I have a related multi-turn RL implementation but it's not that efficient. My wechat is Tianzhe011127.

[quanwei0]

@UbeCc @PeterSH6 @zhaochenyang20 I am working on multi-step RL training for agents and would like to join the wechat group! My wechat id is weiquan0128. Looking forward to connecting and learning more. Thanks!

[UbeCc]

Thank you for your attention! We already have a large group of people working on the feature. Keep syncing if we have any progress!

[XuanRen4470]

same for me. i am also working on multi turn rl . my wechat is x34ren. could you please add me to the group?

[waleko]

@UbeCc @PeterSH6 @zhaochenyang20 I’m excited about multi-turn RL and would be glad to join the group. My WeChat is alex-kovrigin — happy to connect and dive deeper into the topic. Thanks!

[eric-haibin-lin]

demo: https://github.com/volcengine/verl/pull/917

[DachengLi1]

Thank you @eric-haibin-lin! I am curious whether https://github.com/volcengine/verl/pull/917 is ready for use?

[zhaochenyang20]

Thank you @eric-haibin-lin! I am curious whether #917 is ready for use?

indeed please ask us 😂 We get good codes ready. But the validation score still does not get improved. In our close-sourced sandbox, it works. But for open-sourced sandbox, it doesn't work right now. We will open-source and merge it anyway in early next week.

[eric-haibin-lin]

Initial support is available. Please open new issues for specific follow up issues https://verl.readthedocs.io/en/latest/sglang_multiturn/multiturn.html