Issue: gpt3.5-turbo-1106 responds differently from the OpenAI playground

[ziqizhang]

Issue you'd like to raise.

• Langchain version: 0.0.335
• Affected LLM: chatgpt 3.5 turbo 1106 (NOT affecting: gpt 3.5 turbo 16k, gpt4, gpt4 turbo preview)
• Code template, similar to:

llm=ChatOpenAI(model_name='gpt-3.5-turbo-1106', openai_api_key=openai_key ) memory = ConversationBufferMemory() conversation = ConversationChain( llm=llm, memory=memory )

conversation.predict(input='my instructions')

The problem When the identical instructions and parameters are used in the API and the open ai playground, model responds radically different. The open ai playground responds as expected, producing answers. While the API through langchain always returns 'I cannot fufil your request'.

suspected causes Open AI made an upgrade to its API to 1.x, and as far as I know Langchain now only works with open ai 0.28.1. Could that be the reason?

Thanks

UPDATE 13 Nov: I further tested my prompts with OpenAI api 1.2.3 alone, without langchain and everything works fine:

• First I put my prompts into the OpenAI playground, which produced responses
• Second I 'view source' > 'copy', put the python code into a .py file, edited out the last AI response and ran the code = I got the response I wanted.
• Third I tested the same prompt with langchain 0.0.335 (openai 0.28.1) this time, and I always get 'I cannot fulfil this request'.

I know that langchain adds a little bit of context to every request and my prompts did consist of two rounds of chats and thus my code uses memory. I don't know which part of this is problematic.

Again, this problem does not occur with gpt-3.5-turbo, or gpt-4, or gpt-4-turbo, only 'gpt-3.5-turbo-1106'

Suggestion:

No response

[dosubot[bot]]

🤖

Hello @ziqizhang,

I'm an AI here to assist you with your queries and issues related to the LangChain repository while the human maintainers are away. I'm here to help you with debugging, answering your questions, and guiding you on how to contribute to this project.

I'm currently looking into the issue you've reported. I'll provide a detailed response shortly. Thanks for your patience!

[dosubot[bot]]

🤖

Based on the information provided and the context from the LangChain repository, it seems that the discrepancy between the responses from the OpenAI playground and the LangChain API could indeed be due to the recent upgrade of the OpenAI API to 1.x, considering that LangChain currently only works with OpenAI 0.28.1.

This is supported by the code in the completion_with_retry and acompletion_with_retry functions in the libs/langchain/langchain/llms/openai.py file. These functions check if the OpenAI version is 1.x and if so, they call the create method on the client or async_client object. If the OpenAI version is not 1.x, they use a retry decorator to call the create method.

Moreover, the OpenAIChat class in the libs/langchain/langchain/llms/openai.py file, which is used for handling chat models like 'gpt-3.5-turbo-1106', also checks the OpenAI version and initializes the client and async_client objects accordingly. This further suggests that LangChain's handling of OpenAI API calls may differ depending on the version of the OpenAI API, which could lead to discrepancies in the responses from the OpenAI playground and the LangChain API.

In addition, the OpenAIEmbeddings class in the libs/langchain/langchain/embeddings/openai.py file also checks the OpenAI version and initializes the client and async_client objects accordingly. This class is used for generating embeddings from OpenAI models, and its behavior may also differ depending on the version of the OpenAI API. This could potentially contribute to the observed discrepancies as well.

Furthermore, the embed_documents and aembed_documents methods in the OpenAIEmbeddings class, which are used for generating embeddings for a list of texts, do not appear to check the OpenAI version. This could potentially lead to discrepancies if the behavior of the OpenAI API differs between versions when generating embeddings.

Unfortunately, I wasn't able to find the exact version of the OpenAI API that LangChain v0.0.335 is compatible with within the repository. It's possible that this information is available elsewhere or I may have missed it.

I hope this information helps. If you have any further questions or need additional clarification, please let me know.

Sources

• Invalid model when use chain
• libs/langchain/langchain/llms/openai.py
• libs/langchain/langchain/embeddings/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[PavelAgurov]

Hi, I had the same issue - result is fully different between playground and direct call. I resolved it by adding model_kwargs parameters into constructor:

            return ChatOpenAI(
                model_name     = model_name,
                max_tokens     = max_tokens,
                temperature    = 0,
                verbose        = False,
                model_kwargs={
                    "frequency_penalty": 0.0,
                    "presence_penalty" : 0.0,
                    "top_p" : 1.0
                }        
            )

Now it's more or less the same.

[ziqizhang]

Hi, I had the same issue - result is fully different between playground and direct call. I resolved it by adding model_kwargs parameters into constructor:

            return ChatOpenAI(
                model_name     = model_name,
                max_tokens     = max_tokens,
                temperature    = 0,
                verbose        = False,
                model_kwargs={
                    "frequency_penalty": 0.0,
                    "presence_penalty" : 0.0,
                    "top_p" : 1.0
                }        
            )

Now it's more or less the same.

Thank you. I tried this but it still does not work for me... it still outputs 'I cannot fulfil this request' while both the openai playground and the openai 1.2.3 python api produce full response. Frustrating

[dosubot[bot]]

@baskaryan, this one's a bit out of my wheelhouse, could you step in and provide your guidance?