Can we assign scores instead of boolean logic for validation?

[kimianoorbakhsh]

Hello,

I am working on a task that can be evaluated with some score function such as F1-score. I noticed that the validation metrics currently are only boolean (for example, exact match, etc.). Is there a way that I can use numerical metrics? I would appreciate your help and guidance.

Thanks!

[okhat]

Yes you can. See another open issue about the same thing, it has my answer

[okhat]

See issue #145

Does it address what you have in mind or would you like something richer?

[kimianoorbakhsh]

See issue #145

Does it address what you have in mind or would you like something richer?

Thanks for your answer. Yes, but what I was thinking was to use something like a BLEU of F1 score as the metric, since I don't really need the outputs of the model to be exactly the same as their corresponding references in the dataset, but it is acceptable if they are somewhat similar as well.