langchain.llms SelfHostedPipeline and SelfHostedHuggingFaceLLM

[IMJONEZZ]

Hey guys,

I'm trying to use langchain because the Tool class is so handy and initialize_agent works well with it, but I am having trouble finding any documentation that allows me to run this self-hosted locally. Everything seems to be remote, but I have a system that I know is capable of what I'm trying to do.

Is there any way to specify arguments to either SelfHosted class or even runhouse in order to specify that you want it running on the computer you're currently working on, on the gpu connected to that computer, instead of ssh-ing into a remote instance?

Thanks

[tensiondriven]

Only thing I've seen so far is this:

llm = SelfHostedHuggingFaceLLM(model_id="gpt2", hardware=gpu, model_reqs=["pip:./", "transformers", "torch"])

Found here

It looks like SelfHostedHuggingFaceLLM is the class to check out. If you pass it hardware=gpu then it will target your local machine. No idea how to set up the various requirements to stand up the model though; if anyone knows, please comment.

[oapandit]

Hi,

Were you able to do it? I also have similar requirement?

thanks, Onkar

[oapandit]

Only thing I've seen so far is this:

llm = SelfHostedHuggingFaceLLM(model_id="gpt2", hardware=gpu, model_reqs=["pip:./", "transformers", "torch"])

Found here

It looks like SelfHostedHuggingFaceLLM is the class to check out. If you pass it hardware=gpu then it will target your local machine. No idea how to set up the various requirements to stand up the model though; if anyone knows, please comment.

I tried to do this, but got the following error:

ValueError: Cluster must have an ip address (i.e. be up) or have a reup_cluster method (e.g. OnDemandCluster).

[dosubot[bot]]

Hi, @IMJONEZZ! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you were having trouble finding documentation on how to use LangChain locally instead of remotely. tensiondriven suggested using the SelfHostedHuggingFaceLLM class with the hardware=gpu argument to target the local machine. However, oapandit mentioned encountering a ValueError when trying this approach.

Could you please let us know if this issue is still relevant to the latest version of the LangChain repository? If it is, please comment on this issue to let the LangChain team know. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!