stream is only supported for OpenAI LLMs

[mingxin-yang]

from langchain.chat_models import ChatOpenAI

llm_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo"))
index = GPTSimpleVectorIndex.load_from_disk(file_path, llm_predictor=llm_predictor)
response = index.query(query_str, response_mode="compact", streaming=True)

chatopenAI can't use stream, how to supported it?

[logan-markewich]

I think OpenAI needs to enable streaming in their API for chatGPT, but they haven't yet

[mingxin-yang]

I think OpenAI needs to enable streaming in their API for chatGPT, but they haven't yet

@logan-markewich OpenAI's API is support with stream, I used it in node

[mingxin-yang]

nobody face this problem？

[mingxin-yang]

@Disiok @jerryjliu

[neove]

same problem +1

[mingxin-yang]

https://github.com/jerryjliu/llama_index/blob/main/examples/vector_indices/SimpleIndexDemo-ChatGPT.ipynb https://github.com/jerryjliu/llama_index/blob/main/examples/vector_indices/SimpleIndexDemo-streaming.ipynb Hope to use the both example. @jerryjliu 🧐

[xumo97]

Same question

[holmesian]

Same question

[hlwhl]

@mingxin-yang @xumo97 Take this repo as reference https://github.com/hlwhl/knowledge_base_server. It's achieved stream response based on callback manager.

[Disiok]

Closing this ticket to tracker under https://github.com/jerryjliu/llama_index/issues/1007