Issue: How to do token streaming for other LLMs on Streamlit or Gradio

[eshaanagarwal]

Issue you'd like to raise.

Hi i was trying to create a chatbot using LLM and langchain. It would be great if you could point me to any function or method or way to implement token streaming for my chatbot.

I am developing the UI using Streamlit but I can change to gradio too. Is there any specific way to do that? Please tell me.

I need to do it urgently so I would appreciate anybody help.

Suggestion:

No response

[LoveFishoO]

Maybe you can rewrite StreamingStdOutCallbackHandler class

import time
import streamlit as st
from typing import Any, List, Dict
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
class MyStream(StreamingStdOutCallbackHandler):
    
    def __init__(self) -> None:
        super().__init__()
        self.o = st.empty()
        self.s = ''
    
    def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        """Run when LLM starts running."""
        del self.o
        self.s = ''
        self.o = st.empty()
        
    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        """Run on new LLM token. Only available when streaming is enabled."""
        self.s += token
        self.o.info(self.s)
        time.sleep(0.05)

And then use custom class in the callbacks args

LLM = ChatOpenAI(
        temperature = 0,
        streaming=True, 
        callbacks=[MyStream()], 
        )

Hope this demo could give your some inspiration.

[samthedataman]

this solution worked perfectly you are a god!

LoveFishoO

[Saigenix]

Maybe you can rewrite StreamingStdOutCallbackHandler class

import time
import streamlit as st
from typing import Any, List, Dict
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
class MyStream(StreamingStdOutCallbackHandler):
    
    def __init__(self) -> None:
        super().__init__()
        self.o = st.empty()
        self.s = ''
    
    def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        """Run when LLM starts running."""
        del self.o
        self.s = ''
        self.o = st.empty()
        
    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        """Run on new LLM token. Only available when streaming is enabled."""
        self.s += token
        self.o.info(self.s)
        time.sleep(0.05)

And then use custom class in the callbacks args

LLM = ChatOpenAI(
        temperature = 0,
        streaming=True, 
        callbacks=[MyStream()], 
        )

Hope this demo could give your some inspiration.

this does not work with gradio chatInterface, Is there any generator function that langchain provides for streaming

[dosubot[bot]]

Hi, @eshaanagarwal! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about implementing token streaming for a chatbot using LLM and langchain on either Streamlit or Gradio. LoveFishoO provided a solution using Streamlit, which was confirmed to work by samthedataman. However, Saigenix mentioned that the solution does not work with Gradio and asked if there is a generator function provided by langchain for streaming.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project! Let us know if you have any further questions or concerns.