openai-cookbook icon indicating copy to clipboard operation
openai-cookbook copied to clipboard

Add handling for last chunk in split_into_sentences function

Open hunkim opened this issue 2 years ago • 1 comments

I have added handling for the last chunk in the split_into_sentences function. Previously, the function did not account for the last chunk, which could lead to incomplete sentences in the output.

To solve this, I added a conditional statement to check if the last chunk is non-empty. If it is, I append it to the list of chunks with a period to ensure the last sentence is complete.

This change improves the accuracy of the split_into_sentences function and ensures that all sentences in the input text are properly segmented. Please review and let me know if you have any feedback or concerns.

hunkim avatar Feb 19 '23 02:02 hunkim

just for completeness, the same problem also appears in https://github.com/openai/openai-cookbook/blob/main/apps/web-crawl-q-and-a/web-qa.ipynb

peterfabian avatar Mar 08 '23 13:03 peterfabian

Great catch - thank you!

ted-at-openai avatar Mar 16 '23 23:03 ted-at-openai

just for completeness, the same problem also appears in https://github.com/openai/openai-cookbook/blob/main/apps/web-crawl-q-and-a/web-qa.ipynb

@logankilpatrick, do you want to replicate this quick change to the web crawl Q&A notebook when you have time?

ted-at-openai avatar Mar 16 '23 23:03 ted-at-openai