how to use a model loaded from HuggingFace transformers?

[i-am-neo]

@hwchase17 Thanks for sharing this project. I've encountered several challenges in trying to use it and hope you can point me to examples. I haven't found examples in the docs/issues.

1. I'd like to use an LLM already loaded from transformers on a set of text documents saved locally. Any suggestions?

Something like? model_name = "google/flan-t5-large" t5_tokenizer = T5Tokenizer.from_pretrained(model_name) llm = T5ForConditionalGeneration.from_pretrained(model_name, max_length = 500)

2. I'd like to use a custom "search" function for an agent. Can you please share an example? (For what it's worth, I tried FAISS which didn't yield accurate results).

[logan-markewich]

Yea, I second this. Is there no way to use LangChain with a model that is loaded locally?

[logan-markewich]

I think you'll need to create a custom LLM class: https://langchain.readthedocs.io/en/latest/modules/llms/examples/custom_llm.html

One comment to the maintainers if you read this, the documentation could use a bit of help.... very confusing to navigate and figure out what this library can do. It may be off-putting to a lot of users.

But so far I am impressed with the capabilities!

[solyarisoftware]

@i-am-neo Q1 (using a local model)

Right question. As far as I understand, so far you can't, but you can use a model from huggingface hub. See:
https://langchain.readthedocs.io/en/latest/modules/llms/integrations/huggingface_hub.html

[logan-markewich]

You need an API key for HuggingFaceHub to work

If you have an LLM locally already, you can follow the example on the page I linked above

I tried with google-flan... it was meh. Tough to get it to follow instructions in the prompt. Maybe GPT-J would fair better

[solyarisoftware]

@logan-markewich yes, thanks.

As a documentation improvement, maybe the page https://langchain.readthedocs.io/en/latest/modules/llms/examples/custom_llm.html could be improved with an example following what suggested @i-am-neo (an LLM already loaded from transformers)

[logan-markewich]

Yea, the docs leave a lot to be desired lol

Here's a quick example

import torch
from langchain.llms.base import LLM
from transformers import pipeline

class FlanLLM(LLM):
    model_name = "google/flan-t5-xl"
    pipeline = pipeline("text2text-generation", model=model_name, device="cuda:0", model_kwargs={"torch_dtype":torch.bfloat16})

    def _call(self, prompt, stop=None):
        return self.pipeline(prompt, max_length=9999)[0]["generated_text"]

    def _identifying_params(self):
        return {"name_of_model": self.model_name}

    def _llm_type(self):
        return "custom"

[solyarisoftware]

yes, thanks. The documentation has been just improved about it: https://github.com/hwchase17/langchain/issues/936#issuecomment-1422283398

It could be useful to have a notebook that merge your code above here: https://langchain.readthedocs.io/en/latest/ecosystem/huggingface.html#llm

[i-am-neo]

I tried with google-flan... it was meh. Tough to get it to follow instructions in the prompt. Maybe GPT-J would fair better

@logan-markewich Care to share your prompts with Flan? I have the opposite experience.

[dosubot[bot]]

Hi, @i-am-neo! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about using a model loaded from HuggingFace transformers in LangChain. You were looking for examples on how to use a pre-loaded language model on local text documents and how to implement a custom "search" function for an agent. In the comments, users discussed the possibility of using a local model and suggested creating a custom LLM class. They also mentioned the need for improvements in the documentation and provided code examples.

It seems that the issue has been resolved with the documentation being updated and suggestions for code implementation.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!