Validation is very slow during LoRA fine-tuning

Open hey0chen opened this issue 10 months ago • 3 comments

I noticed that i2v inference needs more than 20 minutes during validation, but when I run cli_demo.py it's only need 6 minutes. I don't think that's normal, is there any solution for this?

Feb 19 '25 08:02 hey0chen

The amount of reasoning time depends on the model you are using, the GPU, and the resolution. I need you to provide these details so that I can answer better.

Feb 20 '25 05:02 zRzRzRzRzRzRzR

The amount of reasoning time depends on the model you are using, the GPU, and the resolution. I need you to provide these details so that I can answer better.

Hi, @zRzRzRzRzRzRzR Thanks for your reply, the model I used is CogVideoX-5B, the resolution is 49x480x720 as recommended, the GPU is H100. Howerver, I don't think it had anything to do with my settings.

Feb 20 '25 05:02 hey0chen

During the validation process in training, inference may take longer due to the use of zero-3.

Feb 27 '25 09:02 OleehyO