PainlessInferenceAcceleration icon indicating copy to clipboard operation
PainlessInferenceAcceleration copied to clipboard

Do lookahead and repetition_penalty conflict?

Open zhanweiw opened this issue 2 years ago • 1 comments

After enabled repetition_penalty, will it lower lookahead's probability? If yes, any solution for avoiding the conflict?

zhanweiw avatar Apr 07 '24 02:04 zhanweiw

It indeed may lower the speedup by about 5%-10%. A sufficient warmup could ease the negative effect.

zheyishine avatar May 05 '24 08:05 zheyishine