Testing in a macbook pro

[alonsoir]

Thanks for sharing the model, I have been able to test it on my macbook pro, i9 with 32 GB of ram. I notice that the cpu goes to 400% when inferring the answer, and the gpu goes to 0%. Is it possible to make the model use the gpu? Radeon Pro Vega 20 4GB

[TouchstoneTheDev]

The required package sentence-transformers is available. There is a hyphen instead of underscore in the package name. Currently the model is the quantized version of the mpt-30b-chat model which is being loaded with Ctransformers. You can use the original huggingface llm with this code

import transformers
llm = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-30b-chat',
  trust_remote_code=True
)

You should replace lines 73 to 78 in question_answer_docs.py with the above code. It should work fine ( I haven't tried) with GPU if you have the GPU which can hold the 30b parameter model.

refer this "https://github.com/mayooear/private-chatbot-mpt30b-langchain/issues/1#issuecomment-1614173626"

[alonsoir]

Hi!, i have tried with your suggestion, but i am getting this error:

pip --version pip 23.2.1 from /usr/local/lib/python3.11/site-packages/pip (python 3.11) python3.11 -m pip install --upgrade pip Requirement already satisfied: pip in /usr/local/lib/python3.11/site-packages (23.2.1) python3.11 -m pip install einops Requirement already satisfied: einops in /usr/local/lib/python3.11/site-packages (0.6.1) make server poetry run python question_answer_docs_server.py embeddings_model_name is all-MiniLM-L6-v2 persist_directory is db model_path is models/mpt-30b-chat.ggmlv0.q4_1.bin target_source_chunks is 4 Initializing model for the first time... Loading model... models/mpt-30b-chat.ggmlv0.q4_1.bin This modeling file requires the following packages that were not found in your environment: einops. Run pip install einops Traceback (most recent call last): File "/Users/aironman/git/private-chatbot-mpt30b-langchain/question_answer_docs_server.py", line 139, in init() File "/Users/aironman/git/private-chatbot-mpt30b-langchain/question_answer_docs_server.py", line 127, in init llm=load_model(), ^^^^^^^^^^^^ File "/Users/aironman/git/private-chatbot-mpt30b-langchain/question_answer_docs_server.py", line 58, in load_model llm = transformers.AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aironman/Library/Caches/pypoetry/virtualenvs/private-chatbot-mpt30b-aREpNEbB-py3.11/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 475, in from_pretrained model_class = get_class_from_dynamic_module( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aironman/Library/Caches/pypoetry/virtualenvs/private-chatbot-mpt30b-aREpNEbB-py3.11/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 431, in get_class_from_dynamic_module final_module = get_cached_module_file( ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aironman/Library/Caches/pypoetry/virtualenvs/private-chatbot-mpt30b-aREpNEbB-py3.11/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 306, in get_cached_module_file get_cached_module_file( File "/Users/aironman/Library/Caches/pypoetry/virtualenvs/private-chatbot-mpt30b-aREpNEbB-py3.11/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 268, in get_cached_module_file modules_needed = check_imports(resolved_module_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aironman/Library/Caches/pypoetry/virtualenvs/private-chatbot-mpt30b-aREpNEbB-py3.11/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 151, in check_imports raise ImportError( ImportError: This modeling file requires the following packages that were not found in your environment: einops. Run pip install einops make: *** [server] Error 1

i have tried to install einops, but, as you can see i am using poetry