add "monitor" option to train + learncurve

[NickleDave]

that lets user specify which metric should be monitored on validation step, e.g. accuracy, segment error rate

we want the best segment error rate possible but we only save maximum validation accuracy. This is an issue since a model may have low segment error rate but relatively high frame error at the same time

[yardencsGitHub]

Perhaps it is possible to add this as a hyper parameter?

Something like:

loss_weight_param = alpha [in 0-->1, default=?]

that sets the loss to be: Loss = alpha*FrameErrorRate + (1-alpha)*SegmentErrorRate

[NickleDave]

yes, I agree that it might be useful to provide a combined / compound loss with hyperparameters that weight each of the loss terms. You mean something like this? https://github.com/NickleDave/vak/blob/f99ffe17ebd915b43e7f235ba5ecb127f634619e/src/vak/nn/modules/loss.py#L41

let's discuss when we meet later

but here I'm just talking about the metric that gets monitored on the validation step. Literally the same thing as the monitor parameter for the keras.EarlyStopping callback: https://keras.io/api/callbacks/early_stopping/

[NickleDave]

We would need to change the name of the checkpoint saved on the validation step to something more generic, e.g. checkpoint.best.pt, which would be a breaking change