Result is wrong when running Qwen2-1.5B-Instruct on Intel NPU

[grandxin]

Just follow the example https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/NPU/HF-Transformers-AutoModels/LLM/generate.py

when I use load_low_bit=sym_int4, the result is wrong. -------------------- Output -------------------- <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user 什么是电子竞技<|im_end|> <|im_start|>assistant League League League trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail trail

while using load_low_bit=sym_int8, result is correct. -------------------- Output -------------------- <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user 什么是电子竞技<|im_end|> <|im_start|>assistant 电子竞技（Electronic Sports）是一种以电子游戏为比赛项目，通过网络进行的体育运动。它包括了多种类型的游戏，如《英雄联盟

[plusbang]

Qwen2-7B-Instruct is verified and we will try to reproduce your error first. We will inform you immediately once there is progress.

[plusbang]

Hi, @grandxin , I could not reproduce such error on MTL with 32.0.100.2540 driver.

By using ipex-llm==2.1.0b20240814, the output of Qwen2-1.5B-Instruct with load_low_bit=sym_int4 is

-------------------- Output --------------------
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
什么是电子竞技<|im_end|>
<|im_start|>assistant
电子竞技是一种基于游戏和视频游戏平台的体育运动，参与者使用特定的游戏控制器（如键盘、鼠标、手柄等）进行控制。与其他
--------------------------------------------------------------------------------
done

Please check the driver version and ipex-llm version.

[grandxin]

Hi, @grandxin , I could not reproduce such error on MTL with 32.0.100.2540 driver.

By using ipex-llm==2.1.0b20240814, the output of Qwen2-1.5B-Instruct with load_low_bit=sym_int4 is

-------------------- Output --------------------
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
什么是电子竞技<|im_end|>
<|im_start|>assistant
电子竞技是一种基于游戏和视频游戏平台的体育运动，参与者使用特定的游戏控制器（如键盘、鼠标、手柄等）进行控制。与其他
--------------------------------------------------------------------------------
done

Please check the driver version and ipex-llm version.

ok. have solved, thanks！