intel-extension-for-transformers
intel-extension-for-transformers copied to clipboard
intel
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
When I run: `from intel_extension_for_transformers.transformers.modeling import AutoModelForCausalLM` It triggers this error: > ContextualVersionConflict: (transformers 4.35.2 (/usr/local/lib/python3.10/dist-packages), Requirement.parse('transformers==4.34.1'), {'intel-extension-for-transformers'}) It can be reproduced on Google Colab (CPU runtime). I tried to...
I'm using this code to load the model in 4bit. ``` from transformers import AutoTokenizer, TextStreamer from intel_extension_for_transformers.transformers import AutoModelForCausalLM import torch from datetime import datetime # Hugging Face model_id...
Hi I was trying to run a model with the printing output in the following way and I keep getting what(): unexpectedly reached end of file any idea on how...
Here is the SFTtrainer method i used for finetuning mistral ``` trainer = SFTTrainer( model=peft_model, train_dataset=data, peft_config=peft_config, dataset_text_field=" column name", max_seq_length=3000, tokenizer=tokenizer, args=training_arguments, packing=packing, ) trainer.train() ``` I found different...
The version is relying on transformers 4.34 dependency at the moment, but is requiring transformers 4.35 as per requirement parser. Please fix!
When pip installing this is what i get: Collecting intel-extension-for-transformers Using cached intel-extension-for-transformers-1.2.1.tar.gz (88.4 MB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully....
ModuleNotFoundError: No module named 'cmake' ModuleNotFoundError: No module named 'cpuinfo'
Hi, Two questions please: 1. Do you support multi-prompt batching in any way? I tried via input_ids but the generation failed with "Unsupport multi-batch input-ids": https://github.com/intel/intel-extension-for-transformers/blob/main/intel_extension_for_transformers/llm/runtime/graph/__init__.py#L137 Is there another way?...
When i use python_api_example or streaming_llm python scripts to inference Qwen-14B-Chat，the first two questions were outputted normally, but the third question has been repeating itself since then. I find it...
I'm trying to evaluate the int4 quantized model with the tools using ```python from intel_extension_for_transformers.llm.evaluation.lm_eval import evaluate ``` like what's done in `/examples/huggingface/pytorch/text_generation`. I succeeded when I quantized my local...
