pipecat icon indicating copy to clipboard operation
pipecat copied to clipboard

Add RimeNonJsonTTSService for non-JSON WebSocket API support

Open gokuljs opened this issue 1 month ago • 3 comments

gokuljs avatar Nov 18 '25 21:11 gokuljs

Codecov Report

:x: Patch coverage is 0% with 158 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/services/rime/tts.py 0.00% 158 Missing :warning:
Files with missing lines Coverage Δ
src/pipecat/services/rime/tts.py 0.00% <0.00%> (ø)

... and 8 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Nov 18 '25 21:11 codecov[bot]

#3063

gokuljs avatar Nov 18 '25 22:11 gokuljs

@markbackman, any update here?

gokuljs avatar Nov 26 '25 17:11 gokuljs

@markbackman let me test this, and I will get back to you on this.

gokuljs avatar Dec 08 '25 21:12 gokuljs

@markbackman One of the key observations is that it works correctly in InterruptibleTTSService but not in WebsocketTTSService. I suspect this points to an issue in our clear-message mechanism. In InterruptibleTTSService the client disconnects and reconnects, which is likely why we are not seeing the problem there.

gokuljs avatar Dec 09 '25 16:12 gokuljs

@markbackman One of the key observations is that it works correctly in InterruptibleTTSService but not in WebsocketTTSService. I suspect this points to an issue in our clear-message mechanism. In InterruptibleTTSService the client disconnects and reconnects, which is likely why we are not seeing the problem there.

The InterruptibleTTSService has a _handle_interruption() method:

    async def _handle_interruption(self, frame: InterruptionFrame, direction: FrameDirection):
        await super()._handle_interruption(frame, direction)
        if self._bot_speaking:
            await self._disconnect()
            await self._connect()

This disconnects and reconnects the websocket. Ideally, this is avoided because there are scenarios where the websocket teardown and reconnection take time.

Ideally, we would use the CLEAR message to clear the current generation requests as it's more precise. If that's not possible, then we should remove the sending of the CLEAR message and instead rely on the InterruptibleTTSService base class.

markbackman avatar Dec 09 '25 18:12 markbackman

@markbackman After investigating with the backend team, it turns out the CLEAR message doesn’t actually interrupt the audio, which explains why audio continues after CLEAR. The non JSON path wasn’t designed to perform a full cancellation. Given that, I think the next step is to rely on InterruptibleTTSService for real interruption and remove our _handle_interruption logic.

gokuljs avatar Dec 09 '25 23:12 gokuljs