LangServe: 'invoke' Route Returns Empty Output, While Streaming Works

[VML-Kai]

trafficstars

I'm building a very simple LangChain application that takes as an input a customer feedback string and categorizes it into the following pydantic class:

class AnalysisAttributes(BaseModel):
    overall_positive: bool = Field(description="<sentiment is positive overall>")
    mentions_pricing: bool = Field(description="<pricing is mentioned>")
    mentions_competition: bool = Field(description="<competition is mentioned>")

parser = PydanticOutputParser(pydantic_object=AnalysisAttributes)

Here's how this should work, and it does:

full_pipeline = prompt | model | parser

output = full_pipeline.invoke({"feedback": "This bad company is very expensive."})

expected_output = AnalysisAttributes(overall_positive=False, mentions_pricing=True, mentions_competition=False)
assert output == expected_output.  # this works! :)

This works very well, all good so far! Let's serve it:

app = FastAPI(
  title="LangChain Server",
  version="1.0",
  description="A simple api server using Langchain's Runnable interfaces",
)

pipeline = prompt | model | parser
add_routes(app, pipeline, path="/categorize_feedback")

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="localhost", port=8000)

Now comes the strange part, check this out. On the client side, streaming works:

response = requests.post(
    "http://localhost:8000/categorize_feedback/stream/",
    json={'input': {'feedback': 'Prices are too high.'}}
)
for chunk in response:
    print(chunk.decode())

# event: metadata [...] data: {"overall_positive":false, ...

But the regular invoke does not work, it delivers an empty output:

response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke/",
    json={'input': {'feedback': 'Prices are too high.'}}
)
print(response.json())

# {'output': {}, 'callback_events': [], 'metadata': {'run_id': 'acdd089d-3c80-4624-8122-17c4173dc1ec'}}

I'm pretty sure the problem is the output parser, as if I remove it from the chain, everything works perfectly fine, the invoke as well.

[VML-Kai]

Currently using a workaround using FastAPI directly:

@app.post('/categorize_feedback')
async def categorize_feedback(input_data: FeedbackInput) -> AnalysisAttributes:
    full_pipeline = prompt | model | parser
    output = full_pipeline.invoke(input_data.model_dump())
    return output

I think somehow .invoke isn't being called differently in langserve.

[eyurtsev]

Hello @VML-Kai , thanks for the bug report.

Would you be able to tell me:

1. What are your langserve and langchain versions?
2. Which version of pydantic are you using

---

I'm wondering is whether the automatic type inference is failing when the parser is present and the chain is doing something weird.

If you're able to:

3. Could you share the input and output types of runnable chain? (chain.input_schema.schema() i think or get the jsonschema for those?)
4. Could you manually overload the input/output types:

    def with_types(
        self,
        input_type: Optional[Union[Type[Input], BaseModel]] = None,
        output_type: Optional[Union[Type[Output], BaseModel]] = None,
    ) -> Runnable[Input, Output]:

[eyurtsev]

@VML-Kai I noticed that the URLs that you have a trailing slash

response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke/",
    json={'input': {'feedback': 'Prices are too high.'}}
)```

Could you remove the trailing slash and try again?

```python
response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke",
    json={'input': {'feedback': 'Prices are too high.'}}
)

[danielcj]

@VML-Kai I was experiencing the same thing, but it works as expected when declaring the output type.

Example on your code:

pipeline = prompt | model | parser
add_routes(app, pipeline, path="/categorize_feedback", output_type=AnalysisAttributes)

[eyurtsev]

@danielcj thank you that's helpful! I'll try to reproduce!

[effusive-ai]

I think I'm having this same problem but after some more investigation I'm not sure. Just a simple chain like this

template = "Write a summary of the following article: \"{content}\""
prompt = PromptTemplate(template=template, input_variables=["content"])

add_routes(
    app,
    prompt | llm,
    path="/summary",
)

I'm calling the route from the JS RemoteRunable class. I thought it was just invoke that wasn't working. But now I see both invoke and stream sometimes return a result and sometimes an empty string for the same call. I haven't been able to track down a pattern of why one call works and another doesn't. Just installed langserv yesterday so should be the latest version.

[PhillipsHoward]

Hello all. I have the same issue I think :

Defining this on server side :

(...)
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Chat history
config={"configurable": {"session_id": "test_session"}}
agent_with_history = RunnableWithMessageHistory(
    agent_executor,
    lambda session_id: RedisChatMessageHistory(session_id, url=os.getenv('REDIS_URL')),
    input_messages_key="input",
    history_messages_key="chat_history",
)

app = FastAPI(
  title="LangChain Test Server",
  version="1.0",
  description="A simple API server using LangChain",
)

add_routes(
    app,
    agent_with_history,
    path="/narrator",
)

Then on my "client" side, I try :

def main():
    chat = RemoteRunnable("http://localhost:8000/narrator")
    session_id = generate_random_string(10)
    
    while True:
        human_input = input("You: ")  # Get user input
        response = chat.invoke({'input': human_input}, {'configurable': {'session_id': session_id}})
        print(response)

The response is always an empty {}, while logs in server indicates that everything works fine on server side. (I can see the output displayed in it)

My pip list about this :

langchain                 0.1.5
langchain-cli             0.0.21
langchain-community       0.0.17
langchain-core            0.1.18
langchain-openai          0.0.5
langserve                 0.0.41
fastapi                   0.109.2

Am I missing something ? Thanks in advance !

EDIT :

As @VML-Kai suggested, it works fine with this workaround :

@app.post('/narrator')
async def narrator_chat(input_data : InputData):
    output = narrator_chain.invoke({'input': input_data.input}, {'configurable': {'session_id': input_data.session_id}})
    return output

[realei]

I have the same issue when I wrap Agent as a Langserve endpoint with invoke endpoint

my pip list:

langchain==0.1.14
langchain-cli==0.0.21
langchain-community==0.0.31
langchain-core==0.1.40
langchain-openai==0.1.1
langchain-text-splitters==0.0.1
langserve==0.0.51
langsmith==0.1.40
fastapi==0.110

[Aillian]

is this issue solved? i am facing the same problem

[shreyasjha5706]

i had the same issue, @danielcj 's solution worked for me. Thanks!

[MarkEdmondson1234]

Same issue today with versions:

langchain                      0.2.6
langserve                      0.2.2

/stream endpoint works, /invoke to same chain has empty content:

{"output":{},"metadata":{"run_id":"9c0f41e4-086d-49f3-9d6b-13352e26e6b2","feedback_tokens":[]}}

Tried adding Output type, no change:

class Output(BaseModel):
    content: str
    metadata: Optional[Dict]

chain = chain.with_types(input_type=Question, output_type=Output).pick(["content", "metadata"])

[manova]

I'm having the same issue where stream returns a response but the invoke endpoint returns null. Are there any new findings as to the root cause of this?

[MarkEdmondson1234]

Mine got fixed putting the output class on the app function (eg making the endpoints) not the chain