agents icon indicating copy to clipboard operation
agents copied to clipboard

Add inference ID

Open egoldschmidt opened this issue 5 months ago • 1 comments

This PR reuses the SpeechHandle ID to serve as an inference ID, enabling LLM invocations to be joined against text in the chat log. This allows consumers to commit side effects only once audio is actually played back to the user (vs. being ignored due to interruption, etc).

The approach here is intended to be minimal & backwards compatible. ChatMessage now has an id which is set from SpeechHandle where possible. In the case where it cannot (.say(...) calls, which are invoked before a SpeechHandle is created), we allow callers to supply their own inference ID, if they want.

At LLM invocation time we pass the inference ID through the LLM callback and into the LLM interface itself. (Note: I would love a confirmation that this part of the PR is backwards compatible! Seemed higher risk than the rest).

egoldschmidt avatar Sep 17 '24 02:09 egoldschmidt