PainlessInferenceAcceleration
PainlessInferenceAcceleration copied to clipboard
Do lookahead and repetition_penalty conflict?
After enabled repetition_penalty, will it lower lookahead's probability? If yes, any solution for avoiding the conflict?
It indeed may lower the speedup by about 5%-10%. A sufficient warmup could ease the negative effect.