dmn-ui icon indicating copy to clipboard operation
dmn-ui copied to clipboard

Does including out-of-vocabulary words really have to block the prediction?

Open thundergolfer opened this issue 7 years ago • 1 comments

I was trying to play with this to see the performance of a DMN but I found it pretty hard to write even a short "story" without using an out-of-vocabulary word. Some of the words you can't use are very common like: "has", "it's", "an".

It is possible to let the prediction still run even if out-of-vocabulary words exist? I'd imagine it would degrade performance, but it would make the demo much easier to use.

thundergolfer avatar May 05 '17 07:05 thundergolfer

sorry for a very late response.

using out of vocabulary words is not there as it is out of the scope of Facebook's bAbI dataset is trying to solve. i think it is possible to add support in the backend, but it might lead to unexpected results. The models were trained on very limited vocabulary.

Hrant-Khachatrian avatar Jul 02 '17 11:07 Hrant-Khachatrian