ipex-llm
ipex-llm copied to clipboard
intel-analytics
error with ipex-llm langchain integration for LLAVA model
Hi, I saved the LLAVA model in 4bit using generate.py from: https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/CPU/PyTorch-Models/Model/llava
model = optimize_model(model)
#Added these lines below in the generate.py if SAVE_PATH : model.save_low_bit(save_path_model) tokenizer.save_pretrained(save_path_model) print(f"Model and tokenizer are saved to {save_path_model}")
Then used the low bit saved model in Langchain.
from ipex_llm.langchain.llms import TransformersLLM model_path = <>
print (" loading IPEX_LLM_Model") ipex_llm_model = TransformersLLM.from_model_id( model_id=model_path, model_kwargs={"temperature": 0, "max_length": 1024, "trust_remote_code": True}, )
I am getting this error:
miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32 2024-06-17 18:40:44,748 - WARNING - /home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/langchain/_api/module_import.py:92: LangChainDeprecationWarning: Importing enforce_stop_tokens from langchain.llms is deprecated. Please replace deprecated imports:
from langchain.llms import enforce_stop_tokens
with new imports of:
from langchain_community.llms.utils import enforce_stop_tokens You can use the langchain cli to automatically upgrade many imports. Please see documentation here https://python.langchain.com/v0.2/docs/versions/v0_2/ warn_deprecated(
2024-06-17 18:40:44,750 - WARNING - BigdlNativeLLM has been deprecated, please switch to the new LLM API for sepcific models. loading IPEX_LLM_Model Traceback (most recent call last): File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/ipex_llm/langchain/llms/transformersllm.py", line 138, in from_model_id model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/unittest/mock.py", line 1378, in patched return func(*newargs, **newkeywargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/ipex_llm/transformers/model.py", line 351, in from_pretrained model = cls.load_convert(q_k, optimize_model, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/ipex_llm/transformers/model.py", line 494, in load_convert model = cls.HF_Model.from_pretrained(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1098, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ceed-user/miniforge3/envs/ipex-llm-cpu/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 795, in getitem raise KeyError(key) KeyError: 'llava_llama'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "rag-bigdl-main/rag_multimodal_no_summary.py", line 39, in
What should be the right way to use the saved 4bit LLAVA model in Langchain?
