Chatbot using HuggingFace model, not working

[levalencia]

I am trying to create a chatbot using your documentation from here: https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/chatgpt_clone.html

However, in order to reduce costs, instead of using ChatGPT I want to use a HuggingFace Model. I tried to use a conversational model, but I got an error that this task is not implemented yet and that it only supports TextGeneration and Tex2TextSequence.

So, I went to the hub and got the model ID with the most likes/downloads and I am testing it:

from langchain import PromptTemplate, HuggingFaceHub, LLMChain
from langchain.llms import HuggingFacePipeline
from langchain.llms import HuggingFaceHub
import os
os.environ["HUGGINGFACEHUB_API_TOKEN"] = "x"
from langchain.chat_models import ChatOpenAI
from langchain import PromptTemplate, LLMChain
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)
from langchain import PromptTemplate, HuggingFaceHub, LLMChain
from langchain.memory import ConversationBufferWindowMemory

template = """Assistant is a large language model trained by OpenAI.

Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.

Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.

Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.

{history}
Human: {human_input}
Assistant:"""

prompt = PromptTemplate(
    input_variables=["history", "human_input"], 
    template=template
)

llm = HuggingFaceHub(repo_id="facebook/mbart-large-50", model_kwargs={"temperature":0, "max_length":64})
llm_chain = LLMChain(prompt=prompt, llm=llm, memory=ConversationBufferWindowMemory(k=2))
output = llm_chain.predict(human_input="Whats the weather like?")
print(output)

However in the Google Colab output, all I get is this: Assistant is a large language model trained by OpenAI. Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate

And no prediction is in the output.

What am I missing?

[alexiri]

Wild guess, but could it be because of the max_length you've set?

[levalencia]

I increased the length to 8000 and now I got this:

AI: Assistant is a large language model trained by OpenAI. Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. Human: Whats the weather like? Assistant:

So the answer from Assistant is still empty

[levalencia]

anyone has any additional idea of what am I missing?

[bmwas]

Thanks for sharing this ...looking for a similar solution to cut costs as well. It looks like the issue here is the repo type ---if you change repo to repo_id="google/flan-t5-xl" - you should get a somewhat relevant response. Off course, I'm also trying with more advanced models (esp. dolly-v2) but it's not working for me as well. It also looks like the free api's from huggingface have a problem with loading large models (see - https://discuss.huggingface.co/t/cannot-run-large-models-using-api-token/31844)

[Wamy-Dev]

Apparently Langchain does not support conversation bots yet? I really don't know why that would be though.

[dosubot[bot]]

Hi, @levalencia! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the issue, you were trying to create a chatbot using a HuggingFace model, but encountered an error stating that the conversational task is not implemented yet. You provided code using the HuggingFace model, but the output only returned a partial response from the model. Another user, alexiri, suggested that the issue might be with the max_length parameter. You then increased the length to 8000, but the response from the model was still empty. Another user, bmwas, suggested changing the repo type to repo_id="google/flan-t5-xl" to get a relevant response. Wamy-Dev mentioned that Langchain may not support conversation bots yet.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and cooperation.