Computing train metrics at a given frequency

[Ghelfi]

🚀 Feature Request

Add the possibility to compute metrics on the train dataset with a given frequency and avoid computing these every steps

Motivation

Having the possibility to compute training metrics is great and allow a good control on what is happening at this stage. Some metrics can be quite compute intensive and we might want to compute these on a lower frequency to reduce to compute needed. This should come with a default behaviour falling back to the current one: evaluating the training metrics for each batch.

Implementation

There is a eval_interval argument for the when to run evaluation and I propose to keep the same logic with a train_metric_interval to decide when to compute training metrics.

Happy to have a look and propose an implementation.