Structured Output with Groq: Error code: 400 - {'error': {'message': 'response_format` does not support streaming', 'type': 'invalid_request_error'}}

[weissenbacherpwc]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

    class GradeDocuments(BaseModel):
        score: str = Field(
            description="Die Frage handelt sich um ein Smalltalk-Thema, 'True' oder 'False'"
        )


    def question_classifier(state: AgentState):
        question = state["question"]
        print(f"In question classifier with question: {question}")
        system = """<s>[INST] Du bewertest, ob es in sich bei der Frage des Nutzers um ein Smalltalk-Thema handelt oder nicht. \n
            Falls es bei der Frage um generelle Smalltalk-Fragen wie zum Beispiel: 'Hallo, wer bist du?' geht, bewerte es als 'True'. \n
            Falls es sich bei der Frage um eine spezifische Frage zu einem Thema handelt wie zum Beispiel: 'Nenne mir Vorteile von Multi CLoud' geht, bewerte die Frage mit 'False'.[/INST]"""

        grade_prompt = ChatPromptTemplate.from_messages(
            [
                ("system", system),
                (
                    "human",
                    "Frage des Nutzers: {question}",
                ),
            ]
        )

        #llm = ChatOpenAI() mit ChatOpenAI gehts, mit Chatgroq vorher iwie nicht mehr
        env_vars = dotenv_values('.env')
        load_dotenv()
        groq_key = env_vars.get("GROQ_API_KEY")
        print("Loading Structured Groq.")
        llm = ChatGroq(model_name="mixtral-8x7b-32768", groq_api_key = groq_key)
        structured_llm = llm.with_structured_output(GradeDocuments)
        grader_llm = grade_prompt | structured_llm
        result = grader_llm.invoke({"question": question})
        state["is_smalltalk"] = result.score
        return state

Error Message and Stack Trace (if applicable)

When it comes to grade_llm.invoke:

Error code: 400 - {'error': {'message': 'response_format` does not support streaming', 'type': 'invalid_request_error'}}

Description

Hi,

I want to use a Groq LLM for getting structured output, in my case it should return True or False. The code works fine with using ChatOpenAI() but it fails when using Groq even if it should work with structured output as I have read in the documentation.

I also tried structured_llm = llm.with_structured_output(GradeDocuments, method="json_mode") without success. I also already updated my langchain-groq version.

Does anyone have an idea how to solve this?

EDIT: I also tried with a simple example where it works with ChatOpenAI but not with Groq: With ChatOpenAI:

from langchain_groq import ChatGroq

class GradeDocuments(BaseModel):
    """Boolean values to check for relevance on retrieved documents."""

    score: str = Field(
        description="Die Frage handelt sich um ein Smalltalk-Thema, 'True' oder 'False'"
    )

model = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0) # with this it works
#model = ChatGroq(model_name="mixtral-8x7b-32768", groq_api_key = "")
structured_llm = model.with_structured_output(GradeDocuments)
structured_llm.invoke("Hello, how are you?")
# Returns: GradeDocuments(score='False')

With ChatGroq the same errror like above. But if I use e.g. the Llama 3 model from Groq it works, so seems like it is an issue of the Mixtral 8x7B model

System Info

langchain-groq 0.1.5
langchain 0.2.5