agents
agents copied to clipboard
Add inference ID
This PR reuses the SpeechHandle
ID to serve as an inference ID, enabling LLM invocations to be joined against text in the chat log. This allows consumers to commit side effects only once audio is actually played back to the user (vs. being ignored due to interruption, etc).
The approach here is intended to be minimal & backwards compatible. ChatMessage
now has an id
which is set from SpeechHandle
where possible. In the case where it cannot (.say(...)
calls, which are invoked before a SpeechHandle is created), we allow callers to supply their own inference ID, if they want.
At LLM invocation time we pass the inference ID through the LLM callback and into the LLM interface itself. (Note: I would love a confirmation that this part of the PR is backwards compatible! Seemed higher risk than the rest).