Whisper using prefix shows no speed up

[chiiyeh]

Hi, may I know how is the prefix implemented for faster-whisper? I tried looking at the code, it seems like the tokens will be generated as usual (from the start ignoring the prefix) but if the prefix is used than instead of picking the one with the highest probability the prefix token is picked instead. Original Whisper seems to starting off from where the prefix ends and generating tokens after that. Not sure if my understanding is correct. Initially thought having prefix will speed up the decoding, but doesnt seems to be the case.