[GRPO] Changing QLoRA to LoRA or increasing num_gen does not affect VRAM

[jackswl]

As mentioned in title, for GRPO, changing QLoRA to LoRA didn't affect VRAM

When I change num_gen from 4 to 8, it did not affect any VRAM. When I change 8 to 16, it increased the VRAM by only 4GB. Something seems off here.

[danielhanchen]

We shaved VRAM by a lot, so it's probably correct!

Are you certain on QLoRA / LoRA? load_in_4bit = True / False? That's a weird one - LoRA should use much more VRAM

[jackswl]

Hi @danielhanchen, yup. I changed from QLoRA to LoRA, the VRAM is the same

In fact, I changed my num_gen to 16, and the VRAM also is almost similar to num_gen of 4. I don't know what's the version (I can't access anytime soon), but it should be around 7 days ago when this Issue is posted