Deepgram doesn't seem to always send end of speech events.

[martin-purplefish]

It's a really weird one - it seems like deepgram sometimes doesn't send a final transcript even if I stop speaking. It seems to have to do with background noise.

Note I get this interim transcript with 'Yeah." at 12:26:51

Then I stop talking, and don't get the event. Finally I say "Yeah" again at 12:27:10 (after nearly 20 seconds of mic-muted silence) and then the final transcript comes in.

I dug through the code - this might just be a bug with Deepgram, I don't see where we're filtering out mesages - but it's a really strange one.

{"message": "STT event: {\"type\": \"SpeechStarted\", \"channel\": [0, 1], \"timestamp\": 91.2}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:50.898564+00:00"}
{"message": "STT event: SpeechEventType.START_OF_SPEECH -  alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:50.898662+00:00"}
{"message": "STT event: {\"type\": \"Results\", \"channel_index\": [0, 1], \"duration\": 1.0800018, \"start\": 90.32, \"is_final\": false, \"speech_final\": false, \"channel\": {\"alternatives\": [{\"transcript\": \"Yeah.\", \"confidence\": 0.99609375, \"words\": [{\"word\": \"yeah\", \"start\": 90.96, \"end\": 91.4, \"confidence\": 0.99609375, \"punctuated_word\": \"Yeah.\"}]}]}, \"metadata\": {\"request_id\": \"dfc3c733-fe1f-475f-96a5-f59fef8e4c24\", \"model_info\": {\"name\": \"2-phonecall-nova\", \"version\": \"2024-02-05.31606\", \"arch\": \"nova-2\"}, \"model_uuid\": \"9c7ae805-e600-4e0f-a6a2-725be88b7ede\"}, \"from_finalize\": false}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:51.085481+00:00"}
{"message": "STT event: SpeechEventType.INTERIM_TRANSCRIPT - Yeah. alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:51.085878+00:00"}
{"message": "Interrupted agent speech", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:51.228985+00:00"}
{"message": "VAD end of speech but no final transcript to validate", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:51.361018+00:00"}
{"message": "STT event: {\"type\": \"Results\", \"channel_index\": [0, 1], \"duration\": 2.1800003, \"start\": 90.32, \"is_final\": false, \"speech_final\": false, \"channel\": {\"alternatives\": [{\"transcript\": \"Yeah.\", \"confidence\": 0.9980469, \"words\": [{\"word\": \"yeah\", \"start\": 91.2, \"end\": 91.7, \"confidence\": 0.9980469, \"punctuated_word\": \"Yeah.\"}]}]}, \"metadata\": {\"request_id\": \"dfc3c733-fe1f-475f-96a5-f59fef8e4c24\", \"model_info\": {\"name\": \"2-phonecall-nova\", \"version\": \"2024-02-05.31606\", \"arch\": \"nova-2\"}, \"model_uuid\": \"9c7ae805-e600-4e0f-a6a2-725be88b7ede\"}, \"from_finalize\": false}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:52.155643+00:00"}
{"message": "STT event: SpeechEventType.INTERIM_TRANSCRIPT - Yeah. alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:52.156046+00:00"}
{"message": "STT event: {\"type\": \"Results\", \"channel_index\": [0, 1], \"duration\": 3.1800003, \"start\": 90.32, \"is_final\": false, \"speech_final\": false, \"channel\": {\"alternatives\": [{\"transcript\": \"Yeah.\", \"confidence\": 0.9970703, \"words\": [{\"word\": \"yeah\", \"start\": 91.28, \"end\": 91.78, \"confidence\": 0.9970703, \"punctuated_word\": \"Yeah.\"}]}]}, \"metadata\": {\"request_id\": \"dfc3c733-fe1f-475f-96a5-f59fef8e4c24\", \"model_info\": {\"name\": \"2-phonecall-nova\", \"version\": \"2024-02-05.31606\", \"arch\": \"nova-2\"}, \"model_uuid\": \"9c7ae805-e600-4e0f-a6a2-725be88b7ede\"}, \"from_finalize\": false}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:53.136890+00:00"}
{"message": "STT event: SpeechEventType.INTERIM_TRANSCRIPT - Yeah. alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:26:53.137251+00:00"}
{"message": "VAD end of speech but no final transcript to validate", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.176932+00:00"}
{"message": "STT event: {\"type\": \"Results\", \"channel_index\": [0, 1], \"duration\": 4.279999, \"start\": 90.32, \"is_final\": false, \"speech_final\": false, \"channel\": {\"alternatives\": [{\"transcript\": \"Yeah. Yeah.\", \"confidence\": 0.9970703, \"words\": [{\"word\": \"yeah\", \"start\": 91.28, \"end\": 91.78, \"confidence\": 0.9970703, \"punctuated_word\": \"Yeah.\"}, {\"word\": \"yeah\", \"start\": 94.159996, \"end\": 94.6, \"confidence\": 0.9848633, \"punctuated_word\": \"Yeah.\"}]}]}, \"metadata\": {\"request_id\": \"dfc3c733-fe1f-475f-96a5-f59fef8e4c24\", \"model_info\": {\"name\": \"2-phonecall-nova\", \"version\": \"2024-02-05.31606\", \"arch\": \"nova-2\"}, \"model_uuid\": \"9c7ae805-e600-4e0f-a6a2-725be88b7ede\"}, \"from_finalize\": false}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.256475+00:00"}
{"message": "STT event: SpeechEventType.INTERIM_TRANSCRIPT - Yeah. Yeah. alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.256606+00:00"}
{"message": "STT event: {\"type\": \"Results\", \"channel_index\": [0, 1], \"duration\": 4.4400024, \"start\": 90.32, \"is_final\": true, \"speech_final\": true, \"channel\": {\"alternatives\": [{\"transcript\": \"Yeah. Yeah.\", \"confidence\": 0.9975586, \"words\": [{\"word\": \"yeah\", \"start\": 91.2, \"end\": 91.7, \"confidence\": 0.9975586, \"punctuated_word\": \"Yeah.\"}, {\"word\": \"yeah\", \"start\": 94.32, \"end\": 94.76, \"confidence\": 0.98535156, \"punctuated_word\": \"Yeah.\"}]}]}, \"metadata\": {\"request_id\": \"dfc3c733-fe1f-475f-96a5-f59fef8e4c24\", \"model_info\": {\"name\": \"2-phonecall-nova\", \"version\": \"2024-02-05.31606\", \"arch\": \"nova-2\"}, \"model_uuid\": \"9c7ae805-e600-4e0f-a6a2-725be88b7ede\"}, \"from_finalize\": false}", "level": "WARNING", "name": "livekit.plugins.deepgram", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.464816+00:00"}
{"message": "STT event: SpeechEventType.FINAL_TRANSCRIPT - Yeah. Yeah. alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.465346+00:00"}
{"message": "Validating final transcript", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.465536+00:00"}
{"message": "Validating final transcript", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.465956+00:00"}
{"message": "STT event: SpeechEventType.END_OF_SPEECH -  alternatives", "level": "WARNING", "name": "livekit.agents.pipeline", "pid": 63106, "job_id": "AJ_nRP3NNBXKzM7", "timestamp": "2024-10-16T12:27:10.466586+00:00"}