Memory leak on Ubuntu with Pipecat 0.0.85
pipecat version
0.0.85
Python version
3.12.8
Operating System
Ubuntu 24.04.3 LTS
Use Case Description
I am running pipecat (for a year), with a pipeline of deepgram, openai, elevenlabs through livekit.
Current Approach
This issue occurs only when running in the pods on the cluster.
I can run version 0.0.85 just fine from a macOS.
The memory leak occurs with 0.0.86 and 0.0.92. (Those are the only two other version I've checked.
I incremented from v0.0.80, where I had had it, up one at a time, and .81, .82, .83 and .84 are all fine.
With the exact same code, except for the change in pipecat version, the memory leak occurs.
I updated the deepgram, openai, elevenlabs, and livekit packages from where they were to the latest, and there's still the leak.
Errors or Unexpected Behavior
When I go from pipecat version 0.0.84 to 0.0.85 there is a memory issue that makes the instance increase its memory usage about 3GB/minute.
Additional Context
Pipeline Details Services:
- livekit
- deepgram
- openai
- elevenlabs
- krisp noise filter
- silero vad analyzer
I wonder if anyone here has experienced anything similar, or know what could be happening
There are no known memory leaks in Pipecat. To look into this, you'll need to provide more detailed information on what you're doing and what you're seeing.
I think I located the memory leak to exist in the LiveKitTransportClient.
v0.0.85 introduced video streaming support to LiveKitTransport (via CHANGELOG).
I am using livekit with a video track, and this is why I am seeing the issue. I have written some code to ignore the video track, and it has fixed the issue for me.
I used this prompt with the whole transports.livekit.transport file: Do you see any potential issues in this code that may cause memory issues if a video track was subscribed? and it surfaced this issue.
Issue:
-
_process_video_streamis always started when a video track is subscribed (via_on_track_subscribed_wrapper), and it alwaysputs into_video_queue. - The consumer (
_video_in_task_handler) only runs ifparams.video_in_enabledisTrue.
So if a video track is subscribed and video_in_enabled is False (or your input pipeline is stopped/blocked), _video_queue becomes an unbounded buffer for raw VideoFrameEvents → memory will keep growing as long as the track is active.
Even with video_in_enabled=True, if your downstream processing (conversion, push_video_frame, etc.) can’t keep up with LiveKit’s video rate, the queue can still grow without bound.