Eric Rozell

It's rare that you would use 'test' without 'compare'. We could potentially combine these two commands into a single command.

There may be a new batch evaluation API coming out for testing. This only includes machine trained entities and intents. Add a prototype to leverage the batch API as opposed...

Split Dialogflow, LUIS and Lex into their own repositories

2

We should keep a stable core that does not have dependencies on any specific NLU provider. We could also simplify the docs and the purpose of this library.

Target the overall metric with `*` and any intent or entity with `**`

1

Currently, we only have an option to target the overall intent or entity metric using `*`. Users may want the option to apply the threshold against all intents or entities....

Convert timestamp to opt-in feature

3

Not convinced that we need to do this, but we can spare a few bytes in the NLU results files by only including the timestamp when requested via CLI argument.

Consider adding support for custom parsers of utterances

2

Today, we expect that the utterances JSON file is always an array of utterances with entities in one of two formats, an NLU.DevOps generic format: ```json [ { "text": "order...

Might be useful to add an app setting for a decryption key that can be used to decrypt other app settings values consumed by NLU.DevOps.

Consider removing audio input option for NLU.DevOps

1

My current thinking is that in order to support testing from audio input, we need to look at the end-state of a conversation, not the intermediate state of NLU results....

Allow customization of the LUIS domain

1

We hardcode the LUIS domain: https://github.com/microsoft/NLU.DevOps/blob/master/src/NLU.DevOps.Luis.Shared/LuisTrainClient.cs#L19-L20 We should allow users to target arbitrary LUIS v2 or v3 endpoints.

We have an existing document for testing LUIS with Azure Pipelines, let's create a similar example for GitHub Actions.