Missing GPU support? HuggingFace model too slow

[billy-mosse]

Hi! Amazing work :). I was trying to use this library to test a LLM called GPT-JT but I got a timeout error:

ValueError: Error raised by inference API: Model togethercomputer/GPT-JT-6B-v1 time out

I'm running this in a quite powerful server, and resources shouldn't be an issue. The code is probably not using GPU, but I haven't found a way for making langchain use GPU. Could you please tell me if this is automatic, and if I'm missing something?

Code for reproducibility:

from langchain import PromptTemplate, HuggingFaceHub, LLMChain

template = """Question: {question}

Answer: Let's think step by step."""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm_chain = LLMChain(prompt=prompt, llm=HuggingFaceHub(repo_id="togethercomputer/GPT-JT-6B-v1"))

question = "Is an apple a fruit?"
print(llm_chain.run(question))

[nafizh]

I am actually facing the same problem. Sentencebert models are not using local GPU. I did not see any option to enable GPU in the API.

[thepok]

i have a working prototype with gpu support

https://github.com/hwchase17/langchain/pull/410

[RobinSrimal]

Hi Team, I discovered your work yesterday and it looks really cool. Have you considered integrating Haystack (https://haystack.deepset.ai/overview/intro) into your project? It seems like they are solving some of the issues that you are are working on.

[billy-mosse]

@thepok, would you mind pasting a sample code snippet for using gpu? I want to checkout your branch and start using it

[thepok]

 from langchain.llms import Accelerate
 model_name = "facebook/opt-30b"
 FastLLM = Accelerate.from_model_name(model_name=model_name)
 print(FastLLM("Hello World"))

[francistotle]

I get errors when running the code above

Traceback (most recent call last):
  File "/home/francis/langchain/langchain/llms/accelerate.py", line 110, in from_model_name
    model = cls.get_accelerated_gpt(model_name)
  File "/home/francis/langchain/langchain/llms/accelerate.py", line 54, in get_accelerated_gpt
    from accelerate import (
ModuleNotFoundError: No module named 'accelerate'

[thepok]

chatgtp could have solved that one! "No module named 'accelerate'" you have to install accelerate.

[annabechang]

chatgtp could have solved that one! "No module named 'accelerate'" you have to install accelerate.

could you show/ point me to where i can find how to install accelerate? thank you

[dusvyat]

@annabechang https://huggingface.co/docs/accelerate/basic_tutorials/install

[dosubot[bot]]

Hi, @billy-mosse! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you were experiencing slow performance when using the HuggingFace model in the langchain library. You suspected that the code was not utilizing the GPU and were seeking clarification on whether GPU support is automatic or if you were missing something. Another user, thepok, provided a working prototype with GPU support in a pull request. However, there were some errors reported when running the code, and thepok suggested installing the 'accelerate' module to resolve the issue.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project! Let us know if you have any further questions or concerns.