openai-cookbook
openai-cookbook copied to clipboard
Add handling for last chunk in split_into_sentences function
I have added handling for the last chunk in the split_into_sentences function. Previously, the function did not account for the last chunk, which could lead to incomplete sentences in the output.
To solve this, I added a conditional statement to check if the last chunk is non-empty. If it is, I append it to the list of chunks with a period to ensure the last sentence is complete.
This change improves the accuracy of the split_into_sentences function and ensures that all sentences in the input text are properly segmented. Please review and let me know if you have any feedback or concerns.
just for completeness, the same problem also appears in https://github.com/openai/openai-cookbook/blob/main/apps/web-crawl-q-and-a/web-qa.ipynb
Great catch - thank you!
just for completeness, the same problem also appears in https://github.com/openai/openai-cookbook/blob/main/apps/web-crawl-q-and-a/web-qa.ipynb
@logankilpatrick, do you want to replicate this quick change to the web crawl Q&A notebook when you have time?