Use falcon model in privategpt

[mehrdad2000]

Hi how can i use Falcon model in privategpt?

https://huggingface.co/tiiuae/falcon-40b-instruct

Thanks

[HarihararamSriram]

I believe instead of GPT4All() llm you need to use the HuggingFacePipeline integration from LangChain that allows you to run HuggingFace Models locally.

`from langchain import HuggingFacePipeline

llm = HuggingFacePipeline.from_model_id(model_id="model-id of falcon", task="text-generation")

This will get you any LLM model from huggingface.

https://python.langchain.com/en/latest/modules/models/llms/integrations/huggingface_pipelines.html

[pseudotensor]

Falcon 7B or 40B can be used with h2oGPT right now. The demo without langchain is here: https://falcon.h2o.ai/

Try: https://github.com/h2oai/h2ogpt

You can use our versions from HF that have been instruct tuned:

https://huggingface.co/h2oai/h2ogpt-oig-oasst1-falcon-40b

h2oGPT will automatically use the correct prompt type.

These are Kaggle grandmaster versions for 7B:

https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v2 https://huggingface.co/h2oai/h2ogpt-gm-oasst1-multilang-2048-falcon-7b

For these pass to generate.py also --prompt_type=prompt_answer

Or for OASST falcon versions: https://huggingface.co/OpenAssistant

You just pass to generate.py also --prompt_type=open_assistant

[mehrdad2000]

@pseudotensor how can i use these models xlm-roberta-large-fa-qa and bert-base-fa-qa in h2oai?

[mehrdad2000]

@HarihararamSriram would you please explain more where should i add that line you mentioned?

[HarihararamSriram]

Okay, consider Line 36 of the given code. Here we load the GPT4All model into the variable llm using the GPT4All constructor. Similarly, if you want to use Falcon instead of GPT4All, then the variable llm should be initialized with HuggingFacePipeline.from_model_id(model_id="mtiiuae/falcon-7b", task="text-generation") instead. You can write code for another match-case for Falcon model

match model_type:
        ...
        case "falcon":
            llm = `HuggingFacePipeline.from_model_id(model_id="mtiiuae/falcon-7b", task="text-generation")

This will get you the model from huggingface. Please refer to this link for more information about this HuggingFacePipeline Integration. HuggingFacePipeline

Try and let me know how well it works, because Falcon is a recent model.

[mehrdad2000]

@HarihararamSriram what should i set for ".env" for "MODEL_PATH"? it take bin file, and in huggingface often there are more than single bin file like this: https://huggingface.co/tiiuae/falcon-40b-instruct/tree/main

any idea?

[robinliubin]

falcon works for me in this way: in .env set MODEL_TYPE=falcon in privateGPT.py:

+from langchain import HuggingFacePipeline
+        case "falcon":
+            llm = HuggingFacePipeline.from_model_id(model_id="tiiuae/falcon-7b-instruct",  task="text-generation", model_kwargs={"temperature":0, "max_length":1000,"trust_remote_code": True})

[mehrdad2000]

@robinliubin would you please share your .env? Thanks

[robinliubin]

PERSIST_DIRECTORY=db
MODEL_TYPE=falcon
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2
MODEL_N_CTX=1000
TARGET_SOURCE_CHUNKS=4

[mehrdad2000]

@robinliubin Why your MODEL_PATH still refer to ggml-gpt4all-j-v1.3-groovy.bin?

[robinliubin]

MODEL_PATH is not used at all when you use the HuggingFacePipeline

[SaravgiYash]

@robinliubin But using HuggingFacePipeline won't train the model on private data?

[akashlinux10may]

https://github.com/akashlinux10may/AkashlinuxGPT

Here I have implemented is using falcon. You can refer here.

[jannikstdl]

@akashlinux10may

https://github.com/akashlinux10may/AkashlinuxGPT

Here I have implemented is using falcon. You can refer here.

No you are using a Google HuggingFace model. Also everybody can see your Huggingface API Key in the Repo.

[akashlinux10may]

Thanks for replying and letting me about the API key. You are right there I used flan but same way I tried falcon also there repo id = tiiuae/falcon-7b-instruct

Only problem I am facing that it would also answer the questions out of the context.