WhisperLiveKit icon indicating copy to clipboard operation
WhisperLiveKit copied to clipboard

App crash after 15min of transcription.....

Open minhanh1234 opened this issue 8 months ago • 8 comments

now use why the app completely hang after 15mins of recording. everything seem to working until the 15min mark hit

minhanh1234 avatar Apr 05 '25 00:04 minhanh1234

Confirmed the same. Transcription stops after about 16.5 minutes. No error in the logs, the logs just stop appearing. Also no obvious abnormality in container resource usage like CPU/memory maxing out.

Running without or other options and default model from commit 09279c572afdce7ddffbe54d210f7c317873f406. Also running on GPU but forgot to take a load trace.

Also it is impossible to make a new socket connection after this event.

Image

needabetterusername avatar Apr 11 '25 02:04 needabetterusername

Hi, I am well aware of the problem, I am working on it. btw thank you for the screenshot @needabetterusername

QuentinFuxa avatar Apr 11 '25 10:04 QuentinFuxa

If you need anything let me know!

needabetterusername avatar Apr 11 '25 12:04 needabetterusername

You can try with the latest commit. Except if you have a different problem than mine, the issue is solved

QuentinFuxa avatar Apr 12 '25 13:04 QuentinFuxa

You can try with the latest commit. Except if you have a different problem than mine, the issue is solved

I guess a lot of people cloned this repo reported this issue, and I used your latest commit and I still has same issue, and when I increased 15s in "if ffmpeg_idle_time > 15: " in audio_processor.py it took longer to hang. Anyway I am trying PyAV instead of ffmpeg-python, not sure if I can manage to run it

narges-lux avatar Apr 14 '25 07:04 narges-lux

when I increased 15s in "if ffmpeg_idle_time > 15: " in audio_processor.py it took longer to hang.

Can you share the logs from before and after changing the timeout? ALso can you share the launch config you used?

needabetterusername avatar Apr 15 '25 01:04 needabetterusername

when I increased 15s in "if ffmpeg_idle_time > 15: " in audio_processor.py it took longer to hang.

Can you share the logs from before and after changing the timeout? ALso can you share the launch config you used?

I use

whisperlivekit-server --lan auto --model large-v2 --task translate --backend faster-whisper --buffer_trimming sentence --host 0.0.0.0 --port 61999

and I run it on hpc cluster, log is the same before and after changing the time, only time to crash is diff, also here is the strace on the ffmpeg subprocess made in another screen

Image

narges-lux avatar Apr 15 '25 06:04 narges-lux

https://github.com/QuentinFuxa/WhisperLiveKit/blob/bc7c32100f3df29c5e6d9f81a30ffb5de70737de/whisperlivekit/audio_processor.py#L79

I encountered the same issue — the server consistently crashes around the 16-minute mark. As an initial fix, I changed the following:

pipe_stderr=True → pipe_stderr=False After setting pipe_stderr to False, the server was able to stay alive beyond the 16-minute mark.

With pipe_stderr=False, FFmpeg’s stderr output is now printed directly to the terminal, and I noticed messages appear every few seconds (e.g., size= 51799KB time=00:27...). I suspect that these logs were previously clogging the stderr pipe, eventually causing the server to hang or crash.

Image

treeaaa avatar Apr 17 '25 09:04 treeaaa