neo4j-graphrag-python icon indicating copy to clipboard operation
neo4j-graphrag-python copied to clipboard
Published 2 months ago • verified publisher icon neo4j

Question: How do I search asynchronously using llm response streaming?

Open bldev2473 opened this issue 10 months ago • 2 comments

Hi there.

I'm trying to export the final response of a RAG search to SSE using content streaming from an LLM provider.

How can I integrate the generator response of generate_content_stream() func? The search() function of GraphRAG is synchronous and the invoke() call of the llm model is synchronous.

Image

Do I need to customise GraphRAG?

bldev2473 avatar Feb 22 '25 08:02 bldev2473

Hi @bldev2473 ,

I'm afraid streaming response is not yet possible with this package. The LLMInterface which is responsible for calling the LLM is not managing streaming at the moment.

If this is important to you and you feel like you want to contribute this feature, we'd be happy to support you in the implementation.

stellasia avatar Feb 26 '25 10:02 stellasia

Thank you for your response. I'll use the custom implementation for now and ask for more later.

bldev2473 avatar Apr 14 '25 05:04 bldev2473

Closing as this is not something we are going to work on in the near future. Anyone who want to contribute this, feel free to comment here.

stellasia avatar Jul 04 '25 11:07 stellasia