[Q] Does langchain implement any prompt+response token calculation to prevent hitting max limit

[mahmoudajawad]

When I'm using RetrievalQA with OpenAI and QDrant, I sometimes hit the limit of OpenAI gpt-3.5-turbo of 4097 token limit. Is this expected behaviour, or does langchain implement a tokens calculator that prevents such events?

For the record, I'm hitting this problem with Arabic text, and so far didn't with English, though I don't know if this is correlated or not.

I want to add that in another project I'm working on I have a 100% working token calculator that didn't fail for me in all of my tests (with the focus group working on the project) so far. The way my calculator work is by limiting the initial prompt to max tokens count, then instruct openai SDK to set limit of response to some pre-defined limit (good for three sentences approx.) or the difference between initial prompt tokens count and model tokens limit, whichever is lower.

If this is needed in langchain I'm more than happy to re-implement it for this project with little guidance on where is the best part of the codebase to add this in.

[petmar2017]

same here

[berecat]

same here

[dosubot[bot]]

Hi, @mahmoudajawad! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you raised a concern about whether LangChain has a prompt+response token calculation to avoid exceeding the token limit of OpenAI's gpt-3.5-turbo model. You suggested implementing a token calculator similar to your other project if necessary. It seems that two other users, @petmar2017 and @dreemhai, have also reported experiencing the same issue.

However, the issue is currently unresolved. Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!