supermancmk

PPOV2 Trainner use Deepspeed Zero3 Offload CPU: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu!

3

@younesbelkada @lvwerra @lewtun @kashif @vwxyzjn @edbeeching @qgallouedec @Michellehbn Hi, I use PPOV2 trainer for PPO and run it according to the command given in **examples/scripts/ppo/ppo.py**, but set offload_optimizer_device to CPU...

when running the official gsm8k with tool, multi turn async rollout sglang example without any modifications, the model crashes and appears Nan.

15

I pulled the latest version of verl's code and when running the official gsm8k with tool, multi turn async rollout sglang example without any modifications, the model crashes for training...

Use Megatron for SFT training

1

Could qwen be supported to use Megatron for SFT training? Thanks