Conversation history - limit the number of previous messages included in context

[RylandDeGregory]

Within the Azure OpenAI Studio, you can configure the number of "past messages included" within a conversation.

I've been looking through both the python (/app.py) and typescript (/frontend/src/api/api.ts, frontend/src/pages/chat/Chat.tsx) code, but I can't seem to find any configuration in this sample application that limits the number of previous chat messages within a conversation that are sent to the model as context.

Are all previous messages of a conversation sent into the model, up to the input token limit? And if/when that limit is reached, how does the application handle that?

Thank you!

[sarah-widder]

Hi @RylandDeGregory, yes, all previous messages of the conversation are sent into the model. When the limit is reached, the API will return an error which will surface in the app. Labeling this as an enhancement, it would be a good feature to add a configuration setting for this.

[RylandDeGregory]

Thank you, @sarah-widder! I appreciate the response and agree that it would be a great enhancement.

[johanneskaiser894]

A simple but good implementation for this might be to count the tokens of the context messages before they are sent, and when the threshold of e.g. 8000 is exceeded, the oldest user and assistant messages are deleted (but the system messages are kept). As some messages are shorter and some longer, it is difficult to find a good number of messages to keep in context.

Tiktoken from Openai can be used for the token count with the cl100k_base for gpt-4 and gpt-3.5-turbo models. [https://github.com/openai/tiktoken]