Ngram + Stemmer combination

[ctron]

Using ngram in combination with the stemmer seems to create weird results. Considering the following setup:

Using tantivy: 0.21.0

let ngram = NgramTokenizer::all_ngrams(3, 8).unwrap();
let mut text = TextAnalyzer::builder(ngram)
  .filter(RemoveLongFilter::limit(40))
  .filter(LowerCaser)
  .filter(Stemmer::new(Language::English))
  .build();

Putting in the text September October turns this into:

List of tokens

sep
sept
sept
septem
septemb
septemb
ept
ept
eptem
eptemb
eptemb
eptemb
pte
ptem
ptemb
ptemb
ptember
ptember 
tem
temb
temb
tember
tember 
tember o
emb
emb
ember
ember 
ember o
ember oc
mbe
mber
mber 
mber o
mber oc
mber oct
ber
ber 
ber o
ber oc
ber oct
ber octo
er 
er o
er oc
er oct
er octo
er octob
r o
r oc
r oct
r octo
r octob
r octob
 oc
 oct
 octo
 octob
 octob
 octob
oct
octo
octob
octob
octob
cto
ctob
ctobe
ctober
tob
tobe
tober
obe
ober
ber

I would somehow expect to have this split into September, October and then have the processing on the individual tokens.

[PSeitz]

Do you have a reference for a ngram tokenizer that ends the ngram on whitespace?

RemoveLongFilter::limit(40) doesn't make sense, a ngram token will never have that length. .filter(Stemmer::new(Language::English)) will give unexpected results.

[ctron]

Do you have a reference for a ngram tokenizer that ends the ngram on whitespace?

The example above?

.filter(Stemmer::new(Language::English)) will give unexpected results

Yea, I noticed that :D

Maybe the approach is wrong? Maybe I need something like:

SimpleTokenizer -> RemoveLongFilter -> (Ngram).chain(Stemmer)

I am just not sure how to get there.

[PSeitz]

I meant a reference that does the tokenization in September, October you suggested.

I am just not sure how to get there.

I'm not sure TextAnalzyer can do that currently. You could write you own Tokenizer.

[ctron]

I meant a reference that does the tokenization in September, October you suggested.

That's the SimpleTokenizer one. It gives me:

september
october

[PSeitz]

SimpleTokenizer is not an ngram tokenizer

[ctron]

No it is not. I am sorry, but then I don't understand your question.

[ctron]

So I can guess I can come close to that by somehow reversing the API:

let ngram = NgramTokenizer::all_ngrams(3, 8).unwrap();
let mut text = TextAnalyzer::builder(
    Stemmer::new(Language::English)
        .transform(LowerCaser.transform(RemoveLongFilter::limit(40).transform(SimpleTokenizer::default())))
        .chain(LowerCaser.transform(RemoveLongFilter::limit(40).transform(SimpleTokenizer::default())))
        .chain(LowerCaser.transform(ngram)),
)
.build();

That still gives me things like ber octo, but mostly works.

[adamreichold]

Trying to paraphrase what PSeitz is trying to say: This is the expected behaviour of what is generally called an ngram tokenizer, i.e. it will not care about whitespace. The question for a reference would be some other system, like Lucene/Elasticsearch, which provides such an ngram tokenizer because having a reference would tell us a) such an API part of the state of art and b) give us hints how to add it here if we wanted to.

But indeed in this case, what you are looking for is build your own Tokenizer. Probably by wrapping a TextAnalyzer based on SimpleTokenizer, Stemmer, etc. and then applying NgramTokenizer to the token resulting from that so that you end up with the n-grams of "septemb" and "octob" instead of those of "september october".

[ctron]

Trying to paraphrase what PSeitz is trying to say: This is the expected behaviour of what is generally called an ngram tokenizer, i.e. it will not care about whitespace. The question for a reference would be some other system, like Lucene/Elasticsearch, which provides such an ngram tokenizer because having a reference would tell us a) such an API part of the state of art and b) give us hints how to add it here if we wanted to.

I think there is: https://lucene.apache.org/core/6_6_1/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenFilter.html

The idea we have in mind is to search for sub-words. Like having a text containing SomeHttpConnection, searching for http to find it.

I think it makes sense to combine multiple tokenizers (chaining) to have it split into words (like the Simple one) but then also doing ngrams and stemmers individually on those.

It feels like all of those components are there, but are sometimes implemented a Tokenizers, then as TokenFilter and then the TextAnalyzer. But it doesn't seem to possible to compose the desired behavior as they APIs don't work well together.

Ideally I would want to create some pipeline, like mentioned above.

[adamreichold]

The idea we have in mind is to search for sub-words. Like having a text containing SomeHttpConnection, searching for http to find it.

In that case, you might want to look at SplitCompoundWords which will split based on user-supplied dictionary. This can be more efficient compare to the more brute-force approach of using n-grams but its success depends entirely on the quality of the dictionary.

(In this particular example, you might actually want to build a TokenFilter that splits camel case identifiers but I am not sure whether this encompasses your whole use case.)

[ctron]

In that case, you might want to look at SplitCompoundWords which will split based on user-supplied dictionary. This can be more efficient compare to the more brute-force approach of using n-grams but its success depends entirely on the quality of the dictionary.

Unfortunately, we don't have a dictionary. So that' doesn't really work well.

(In this particular example, you might actually want to build a TokenFilter that splits camel case identifiers but I am not sure whether this encompasses your whole use case.)

I guess that would actually be one way to deal with this. I think it would be great to have more tooling around composing tokenizers and filters. I raised a PR to chain two (or more) tokenizers: https://github.com/quickwit-oss/tantivy/pull/2304 … I believe that's generic enough.