[Question] Loss Factor to Weight LoRA Loss and VSD Loss

[zelingao98]

Thank you for your awesome research!!!

I have been trying to implement prolific dreamer recently. Would you like to share more details about how to set factors for VSD loss and LoRA loss here.

Best, Kuma

[thuwzy]

Thank you for your interest in our work! The factor of VSD loss and LoRA loss does not matter, since they optimize different parameters. To balance these two loss, setting the learning rates for them is important.