starcoder2 icon indicating copy to clipboard operation
starcoder2 copied to clipboard

Published 20 hours ago • verified publisher icon bigcode-project

support SPM mode for FIM prompts

Open erfanium opened this issue 1 year ago • 0 comments
trafficstars

from fim paper (https://arxiv.org/pdf/2207.14255.pdf) section 3.1: SPM mode can be used to reuse kv cache across completion requests.

SPM modes can enable further latency optimization (which is very important in case of code completion tools). is there any reason that startcoder models are using normal PSM mode?

erfanium avatar Mar 13 '24 11:03 erfanium