[Feature] how to calculate validation loss

[clytze0216]

Motivation

I would like to inquire about how to compute the validation loss as shown in Figure 2 of your VSA paper. Specifically, could you provide the training pipeline code snippet that calculates the validation loss? Any insights or code examples you can provide would be greatly appreciated. Thank you!

Related resources

No response

[BrianChen1129]

We just use a small portion(1-5%) of the training dataset as a validation set and compute the loss in the same way as the training. This has not yet been implemented in the FastVideo repository, but we plan to add it soon.

[EricLina]

Hi, @BrianChen1129

I tried to use the finetune scripts on Wan1.3B with Syn-data, but I found that the training loss nearly doesn't decrease throughout the process. Is this normal? Are there common reasons or recommended troubleshooting steps for this behavior?

Thank you!

[EricLina]

With part of Syn-data (400), I tuned it with about 1k steps

[SolitaryThinker]

I found that the training loss nearly doesn't decrease throughout the process

Yes this is normal for diffusion models

[EricLina]

We just use a small portion(1-5%) of the training dataset as a validation set and compute the loss in the same way as the training. This has not yet been implemented in the FastVideo repository, but we plan to add it soon.

Any updates? @BrianChen1129