[Feature request / help] Evaluate on different dataset to training dataset

[Peter-Devine]

Reminder

• [X] I have read the README and searched the existing issues.

Reproduction

I would like to know if there is an option to evaluate the model on a different dataset to the training dataset.

For example, I would like to train on MADLAD and evaluate my model at every evaluation epoch on UltraChat-200K. Is this possible?

If not, could this be implemented in a future release?

Thanks!

Expected behavior

No response

System Info

No response

Others

No response

[Katehuuh]

I would like to know if there is an option to evaluate the model on a different dataset

Similar to request #2616