Guancheng Fu comments

Results 36 comments of


                                            Guancheng Fu

trafficstars

Performance drop for neural-chat 7b with new repo of ipex-llm(2.5.0b20240425) vllm serving.

The image you should use is this one: `intelanalytics/ipex-llm-serving-xpu` Try check the README here: https://github.com/intel-analytics/ipex-llm/tree/main/docker/llm/serving/xpu/docker

Performance drop for neural-chat 7b with new repo of ipex-llm(2.5.0b20240425) vllm serving.

> > The image you should use is this one: `intelanalytics/ipex-llm-serving-xpu` > > Try check the README here: https://github.com/intel-analytics/ipex-llm/tree/main/docker/llm/serving/xpu/docker > > Can we remove the deprecated docker image? Yes, we...

IPEX-LLM on Intel Max Series 1100 for inference libintel-ext-pt-gpu.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

The issue https://github.com/intel/intel-extension-for-pytorch/issues/317 was caused by self-built torch with pre-built intel-extension-for-pytorch. I have tried install these two dependencies in conda and then `import intel_extension_for_pytorch as ipex`. Everything works fine. Can...

IPEX-LLM on Intel Max Series 1100 for inference libintel-ext-pt-gpu.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

After some investigation, the issue was caused not installing the correct torchaudio version. Install using these instructions: ```bash pip install TTS pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ # Then...

IPEX-LLM on Intel Max Series 1100 for inference libintel-ext-pt-gpu.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

The code you provided is using **ipex**, not **ipex-llm**. Please do check again. The intel-extension-for-pytorch repo address is here: https://github.com/intel/intel-extension-for-pytorch Your related code snippet: ```python import intel_extension_for_pytorch as ipex synthesizer.tts_model...

Integrate IPEX-LLM into serve worker

@infwinston Hi, great thanks for maintaining this fantastic project. Any thoughts on this integration PR? :smiley:

vLLM offline_inference.py failed to run on CPU inference

Hi, the vLLM CPU backend is removed for now. The support will be added back later. Sorry for the inconvenience.

Slow text generation on dual Arc A770's w/ vLLM

Hi, I am trying to reproduce this issue in my environment. Will update to this thread.

Slow text generation on dual Arc A770's w/ vLLM

Hi, could you tell me how you tested the performance mentioned in the thread(around 14-15t/s and 8-9t/s, respectively)? Also could you share the results of this script? https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/scripts/env-check.sh

Slow text generation on dual Arc A770's w/ vLLM

Hi, can you check if you have installed the out-of-tree driver on the host? You can check it through the following command: ```bash apt list | grep i915 # WARNING:...