Retrieve term positions within documents

[shachibista]

Hi!

Is there any way to retrieve term positions from the documents? I want to implement a term-highlighting functionality. Is there something for this already built-in?

Thanks!

[shachibista]

A possible solution seems to get the term positions using the Index Reader API and use the analyser to get the offsets of the terms in the document.

from pyserini.analysis import get_lucene_analyzer
from pyserini.pyclass import autoclass

JOffsetAttribute = autoclass('org.apache.lucene.analysis.tokenattributes.OffsetAttribute')

analyser = get_lucene_analyzer()
text = "City buses are running on time"
ts = analyser.tokenStream(None, text)

ts.reset()
while ts.incrementToken():
    offset = ts.attributes.get(JOffsetAttribute)

    start = offset.startOffset()
    end = offset.endOffset()

    print((start, end, text[start:end]))
ts.end()
ts.close()

[lintool]

Hi @shachibista - thanks for your issue, hope you're finding Pyserini useful.

Looping in @ola13 - which faced exactly the same issue recently... thoughts?

[ola13]

Hi @shachibista! I'm working on open sourcing my server code, will answers with links to that code in a day or two

[ola13]

Hi - so - I'm using this code to get highlighted terms https://github.com/huggingface/roots-search-tool/blob/main/web/server.py#L101-L103

What it does is the following:

1. use analyzer to get query terms
2. use analyzer to get document terms
3. save each word from a document which matches a query term
4. highlight anything in that list

The problem with this solution is that I don't actually get any feedback on which term was matched in a given document, I highlight any possible match instead - if that makes sense?