OpenVoiceChat icon indicating copy to clipboard operation
OpenVoiceChat copied to clipboard

Premove LLM request

Open fakhirali opened this issue 1 year ago • 0 comments

Send the LLM request before the silence is completely detected. For example if the silence seconds is 2 s, send an LLM request with all the available transcription after 1s of silence. If the silence was indeed for 2 seconds we'll save 1s on LLM request time, if there was no silence and the user started speaking again we lose money on some tokens.

Should be a param.

fakhirali avatar Oct 10 '24 01:10 fakhirali