fix iteration error

[justanotherlad]

trafficstars

array iteration error: sentences up till max_len of token exceeds should be returned

Here's a screenshot of before and after the change (vertically in terminal) in the code. Clearly, the below should be returned: (The numbers vertically represent current tokens and total_tokens number for each iteration.

[ted-at-openai]

Great catch and thanks for the super helpful documentation.

Would it be easier just to write sentences[:i] instead of sentences[:i+1][:-1]? Seems odd that the list is being sliced twice in the first place.

But happy to merge as is if you don't feel like doing any extra work. Your work so far is already appreciated.

[justanotherlad]

Sure, done. That sounds more convenient unless, as I previously thought, some edge cases were trying to be handled.

[justanotherlad]

Also, please see my comment on Document Library Pre-Processing #79 . I think there might be a bug there.

[justanotherlad]

@ted-at-openai Do you want to close/merge this PR?

[ted-at-openai]

Thanks! Merged.