haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Avoid truncates sentences in context window

Open jcnewell opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. The context window provided in Q&A results typically starts and ends mid-way through a sentence. This makes it unsuitable to present to users.

Describe the solution you'd like The solution is to only include complete sentences in the context window.

Describe alternatives you've considered Matching the current context window with the body text and then extracting the required sentences is inefficient and inconvenient.

Additional context The context window is potentially more useful to users than the answer field itself.

jcnewell avatar Sep 11 '23 13:09 jcnewell

Hey sorry for the late reply. I guess you mean the context_window_size parameter of Extractive QA answers?

Honestly, this is just a helper, you can always get the full context by looking at the Document where the answer came from. Those Documents are the result of splitting files into smaller chunks, and you can split those chunks on sentence level by setting the split_respect_sentence_boundary=True in our PreProcessor

Timoeller avatar Oct 09 '23 09:10 Timoeller