About asynchronous generation and training like AReal

[llv22]

One question about ART framework, will we plan to support asynchronous generation/rollout and training, like https://github.com/inclusionAI/AReaL?tab=readme-ov-file (paper: https://arxiv.org/pdf/2505.24298)?

Essentially, it is a non-blocking rollout mechanism so that the ready-to-use rollout group can be sent to training queue, instead of waiting (If I understand correctly)

Best Regards Orlando

[saum7800]

Hey Orlando!

We have a helper for doing Async RL. check out art.trajectory_group_batches(). example of using it can be found in https://github.com/OpenPipe/ART/blob/aab795fe3fcede8b69744e7b1e43f6fe5f4961f9/dev/tau-bench/run_rl.py#L247 . However, we have seen some stability issues during training due to rollouts being off policy, but i'd love to know what results you see on your tasks!

Let me know if you meant something else here

[llv22]

@saum7800 Thanks a lot for your quick response. Will try to apply async RL and let you know the result ^-^. We now run our experiment on H200, rollout/training=3/1 somehow