Change NLU.DevOps.ModelPerformance to generic JSON compare
Change NLU.DevOps.ModelPerformance to a configurable JSON compare library such that any two JSON values can be compared.
We could have a simple interface like IConfusionMatrixEvaluator:
public interface IConfusionMatrixEvaluator
{
IEnumerable<TestCase> Evaluate(JToken expected, JToken actual);
}
E.g., for NLU today, we would create the following config:
{
"intent": "default",
"text": "string-no-punctuation",
"entities": "NLU.DevOps.ModelPerformance.Comparers.EntitiesEvaluator"
}
Where EntitiesEvaluator is an implementation of IConfusionMatrixEvaluator and string-no-punctuation and default is syntactic sugar for other evaluator implementations.
We may want to have an option that allows you to evaluate all JSON properties in the expected and actual JSON values (not just the configured properties), in which case any unconfigured property value would just use the default evaluator.
Turns out this is likely more challenging than I originally thought. Seems for NLU at least, that we need a custom comparer specific to NLU results for each property (intent, text and entities).
Probably a better idea is to use a prioritized list of comparers with target jpath queries.