ipex-llm icon indicating copy to clipboard operation
ipex-llm copied to clipboard
verified publisher icon intel-analytics

starcoder2 optimization results

Open RobinJing opened this issue 1 year ago • 1 comments

With the starcoder2-3B the 2nd+ token latency is not well-performed, do you have any ideas about it? Thanks!

RobinJing avatar Apr 07 '24 07:04 RobinJing

Hi @RobinJing,

We recently applied further optimization to starcoder2-3b. Please have a try again with ipex-llm >= 2.1.0b20240405 and let us know for any further problems :)

Oscilloscope98 avatar Apr 07 '24 07:04 Oscilloscope98