Do we need truncate strategy options for GPT/Completions?

[mrsteele]

Currently the module removes them in pairs. This is because the assumed format is as follows:

  system: You are a bot...
+ user: Hello, how are you?
- assistant: I am good, how about yourself?
+ user: Good. I have a question
- assistant: What is your question?
+ user: Can I have a pony?

You will see that if we had a strict limit above, the context would be well maintained by removing the earliest pair like so:

  system: You are a bot...
+ user: Good. I have a question
- assistant: What is your question?
+ user: Can I have a pony?

But I'm curious if this doesn't fit everyone's use-case. Any feedback would be appreciated if this doesn't work for your intended use of the module.

(I used this cheat sheet to help with colors for anyone else interested)

[mrahmadt]

I think this is good

I was worried about function messages but after thinking about it, function messages are not worth to be included in the history

Another strategy that could be even better as another option in the library is to use openAI to summarize the conversation

Example if the history reach 2000 token the. Send the history to OpenAI and ask it to summarize the conversation to x words and to keep only the useful information in the history (maybe even to return bullet points of important information in the history)

[mrsteele]

Good thought @mrahmadt . Glad you and I are both thinking about this 😉