BESSER-Bot-Framework icon indicating copy to clipboard operation
BESSER-Bot-Framework copied to clipboard

Inconsistent Training / Faulty recognition of exact sentence

Open Aran30 opened this issue 1 year ago • 0 comments

While working on a bot model, I noticed two things.

First of all, each restart of the bot (and thus each re-training of the NLP model) causes the recognition scores to variate.

While not necessarily bad, in some examples this might create inconsistencies in how bots work.

Secondly, and linked to the first "problem", the wrong intent is recognized even though the exact word or sentence that is present in the training data for the correct intent is given as input.

The example in question:

stop_entity = bot.new_entity('stop_entity', entries=bus_stops)
city_entity = bot.new_entity('city_entity', entries=cities)
stop_intent = bot.new_intent('stop_intent',[
    'STOP'
])
city_intent = bot.new_intent('city_intent',[
    'CITY'
])

"cities" is a list of cities from Luxembourg and "bus_stops" is the list of bus stops in Luxembourg.
This might lead to "cities" containing an example like "Differdange" and "bus_stops" examples such as "Differdange, Schoul" or "Differdange, Fousbann". While it is understandable that the NLP engine might not entirely recognize the "city_intent" with a 100% confidence, I feel that when writing the exact sentence "Differdange, Fousbann" that the "stop_intent" should be recognized. Yet, currently, depending on the restart, "Differdange, Fousbann" will be classified as "city_intent"

Any thought, correction or opinion is appreciated.

Aran30 avatar Sep 19 '23 08:09 Aran30