Extend KFold and Blind tests to show benefit of disambiguation

[acpang]

Detail For clients who are using standard and not plus, provide output to be able to demonstrate if or to what extent disambiguation would help them.

With disambiguation, return the top intents (provided the nodes are configured to do this and the intents are within 55% of the top intent). Based on the output if a good percentage of utterances classified as wrong by the top intent would still have gotten the user to the right answer via disambiguation, that would be a compelling reason to upgrade.

Benefits:

• How many questions are answered correctly by the top intent (current test already provide this)
• How many times was a disambiguation choice presented
• How many times was a correct answer included in the disambiguation choices

High Level Tasks:

• Hit the message api with multiple intents set to true
• Calculate the top intent percentage to determine if the next highest intents detected would have been presented in a disambiguation scenario.
• The Disambiguation threshold (55%) may be configurable in the future. Allow this to be easily modified.

Example of output: Disambiguation Benefit Analysis.xlsx

[acpang]

Creating a separate report to show this detail.

[acpang]

Github dev branch - https://github.com/cognitive-catalyst/WA-Testing-Tool/tree/117_disambiguation_benefits

New File outputs to kfold and blind: test-out-extended.csv test-out_disambiguation.csv

Config.ini.sample new inputs: disambiguation_threshold - (1-99) - default to 55 disambiguation_max_intents - (1-9) - default to 5

[andrewrfreed]

Disambiguation is available on lite plans now.