pipecat icon indicating copy to clipboard operation
pipecat copied to clipboard
verified publisher icon pipecat-ai

Potential Memory Leak in with `videotestsrc` `GStreamerPipelineSource`

Open ken-kuro opened this issue 8 months ago • 2 comments

pipecat version

0.0.63

Python version

3.13.3

Operating System

Fedora 42

Issue description

When using in a Pipecat pipeline, a memory leak is observed over time, even with a simple GStreamer pipeline like videotestsrc. This suggests a potential issue with resource management within the processor or its interaction with GStreamer/Pipecat's event loop. GStreamerPipelineSource

Reproduction steps

  1. Setup the pipecat project to run examples
  2. Run the 18a-gstreamer-videotestsrc.py example (maybe you should use ulimit -v MEM_LIMIT first to prevent actual memory shortage)

Expected behavior

The memory usage of the process should remain relatively stable after initial startup, fluctuating within a small range.

Actual behavior

The memory usage of the Python process steadily increases over time, indicating a memory leak.

Logs

➜ ulimit -v 5242880                                                                                                                                                                                                  at [11:33:34 AM] 
➜ python 18a-gstreamer-videotestsrc.py                                                                                                                                                                               at [11:33:35 AM] 
2025-04-28 11:33:35.488 | INFO     | pipecat:<module>:13 - ᓚᘏᗢ Pipecat 0.1.dev3503 ᓚᘏᗢ
Looking for dist directory at: /home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib/python3.13/site-packages/pipecat_ai_small_webrtc_prebuilt/client/dist
2025-04-28 11:33:36.068 | INFO     | run:main:187 - Successfully loaded bot from /home/ken-kuro/PycharmProjects/pipecat/examples/foundational/18a-gstreamer-videotestsrc.py
2025-04-28 11:33:36.069 | INFO     | run:main:190 - Detected WebRTC-compatible bot, starting web server...
INFO:     Started server process [55993]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:7860 (Press CTRL+C to quit)
2025-04-28 11:33:41.351 | DEBUG    | pipecat.transports.network.webrtc_connection:_initialize:128 - Initializing new peer connection
2025-04-28 11:33:41.360 | DEBUG    | pipecat.transports.network.webrtc_connection:_create_answer:206 - Creating answer
2025-04-28 11:33:41.361 | DEBUG    | pipecat.transports.network.webrtc_connection:on_track:189 - Track audio received
2025-04-28 11:33:41.362 | DEBUG    | pipecat.transports.network.webrtc_connection:on_track:189 - Track video received
2025-04-28 11:33:41.363 | DEBUG    | pipecat.transports.network.webrtc_connection:on_icegatheringstatechange:185 - ICE gathering state is gathering
2025-04-28 11:33:46.364 | DEBUG    | pipecat.transports.network.webrtc_connection:on_icegatheringstatechange:185 - ICE gathering state is complete
2025-04-28 11:33:46.365 | DEBUG    | pipecat.transports.network.webrtc_connection:_create_answer:209 - Setting the answer after the local description is created
INFO:     ::1:43652 - "POST /api/offer HTTP/1.1" 200 OK
2025-04-28 11:33:46.366 | INFO     | 18a-gstreamer-videotestsrc:run_bot:24 - Starting bot with video test source
2025-04-28 11:33:46.387 | DEBUG    | pipecat.processors.frame_processor:link:177 - Linking PipelineSource#0 -> GStreamerPipelineSource#0
2025-04-28 11:33:46.387 | DEBUG    | pipecat.processors.frame_processor:link:177 - Linking GStreamerPipelineSource#0 -> SmallWebRTCOutputTransport#0
2025-04-28 11:33:46.387 | DEBUG    | pipecat.processors.frame_processor:link:177 - Linking SmallWebRTCOutputTransport#0 -> PipelineSink#0
2025-04-28 11:33:46.387 | DEBUG    | pipecat.processors.frame_processor:link:177 - Linking PipelineTaskSource#0 -> Pipeline#0
2025-04-28 11:33:46.387 | DEBUG    | pipecat.processors.frame_processor:link:177 - Linking Pipeline#0 -> PipelineTaskSink#0
2025-04-28 11:33:46.387 | DEBUG    | pipecat.pipeline.runner:run:38 - Runner PipelineRunner#0 started running PipelineTask#0
2025-04-28 11:33:46.387 | INFO     | pipecat.transports.network.small_webrtc:connect:306 - Connecting to Small WebRTC
2025-04-28 11:33:46.388 | DEBUG    | pipecat.transports.network.webrtc_connection:on_iceconnectionstatechange:179 - ICE connection state is checking, connection is connecting
2025-04-28 11:33:46.388 | DEBUG    | pipecat.transports.network.webrtc_connection:_handle_new_connection_state:305 - Connection state changed to: connecting
2025-04-28 11:33:46.412 | DEBUG    | pipecat.transports.network.webrtc_connection:on_iceconnectionstatechange:179 - ICE connection state is completed, connection is connecting
2025-04-28 11:33:46.414 | DEBUG    | pipecat.transports.network.webrtc_connection:_handle_new_connection_state:305 - Connection state changed to: connected
2025-04-28 11:33:46.414 | DEBUG    | pipecat.transports.network.small_webrtc:on_connected:166 - Peer connection established.
2025-04-28 11:33:46.414 | DEBUG    | pipecat.transports.network.webrtc_connection:replace_video_track:270 - Replacing video track video
WARNING:aiortc.rtcrtpsender:RTCRtpsender(video) Traceback (most recent call last):
  File "/home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib64/python3.13/site-packages/aiortc/rtcrtpsender.py", line 359, in _run_rtp
    enc_frame = await self._next_encoded_frame(codec)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib64/python3.13/site-packages/aiortc/rtcrtpsender.py", line 286, in _next_encoded_frame
    data = await self.__track.recv()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib/python3.13/site-packages/pipecat/transports/network/small_webrtc.py", line 129, in recv
    frame = VideoFrame.from_ndarray(frame_data, format="rgb24")
  File "av/video/frame.pyx", line 746, in av.video.frame.VideoFrame.from_ndarray
  File "av/video/frame.pyx", line 117, in av.video.frame.VideoFrame.__cinit__
  File "av/video/frame.pyx", line 136, in av.video.frame.VideoFrame._init
  File "av/error.pyx", line 424, in av.error.err_check
av.error.MemoryError: [Errno 12] Cannot allocate memory

Traceback (most recent call last):
  File "/home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib/python3.13/site-packages/pipecat/processors/gstreamer/pipeline_source.py", line 205, in _appsink_video_new_sample
    image=info.data,
          ^^^^^^^^^
MemoryError
Traceback (most recent call last):
  File "/home/ken-kuro/PycharmProjects/pipecat/examples/foundational/.venv/lib/python3.13/site-packages/pipecat/processors/gstreamer/pipeline_source.py", line 205, in _appsink_video_new_sample
    image=info.data,
          ^^^^^^^^^
MemoryError

(python:55993): GLib-ERROR **: 11:33:53.301: ../glib/gmem.c:106: failed to allocate 2764943 bytes
fish: Job 1, 'python 18a-gstreamer-videotests…' terminated by signal SIGTRAP (Trace or breakpoint trap)

ken-kuro avatar Apr 28 '25 04:04 ken-kuro

@aconchillo please take a look when you have some time.

markbackman avatar Apr 30 '25 03:04 markbackman

Hi there, I've been looked into this issue for a few days now. I am observing significant memory consumption issues and resources leaks with GStreamerPipelineSource connected to SmallWebRTCOutputTransport, though the behaviour differs slightly depending on the GStreamer source, it does points towards problem with backpressure handling, resource cleanup, and maybe state management between connections.

There're two main scenarios:

  1. High-rate source (or the videotestsrc):
  • Setup: videotestsrc ! capsfilter caps="video/x-raw,width=1280,height=720,framerate=30/1,format=RGB" -> SmallWebRTCOutputTransport
  • Behavior: Memory usage increases extremely rapidly (multiple GBs in seconds), quickly leading to OOM erros or crashes. This seems related to the high data rate (~79 MB/s), overwhelming the WebRTC processing/encoding pipeline.
  1. Finite Source (filesrc) & Reconnects:
  • Setup: filesrc location=<video.mp4> ! decodebin ! ... -> SmallWebRTCOutputTransport
  • Behavior:
    • During the first connection, the video plays correctly.
    • Memory increases even after the file source has finished playing (EOS reached).
    • After disconnecting the WebRTC client and attempting to reconnect, subsequent pipeline runs fail to output video.
    • On these subsequent connection attempts, resource exhaustion errors appear in the logs, such as:
      • GStreamer-WARNING **: failed to create thread: Error creating thread: Resource temporarily unavailable
      • RuntimeError: can't allocate lock
      • MemoryError (in Python)
      • GLib-ERROR **: ... failed to allocate memory
      • Process eventually crashes (SIGABRT).

Hypotheses and Analysis:

  • High-Rate Source Issue: The primary hypothesis is that the ~79MB/s raw RGB data rate massively exceeds the processing/encoding capacity of aiortc. This leads to frames accumulating in internal, unbounded asyncio.Queues, specifically identified as potentially being BaseOutputTransport._video_out_queue and RawVideoTrack._video_buffer. Lack of effective backpressure propagation allows the source to keep pushing data until memory is exhausted.
  • Finite Source & Reconnect Issue: The behavior (memory increase after EOS, failure on reconnect, resource errors) strongly suggests incomplete resource cleanup during pipeline shutdown/disconnect.
    • Hypothesis: GStreamer resources (threads, bus watches, pipeline/element objects), aiortc resources (peer connection, tracks, background tasks), and/or Pipecat asyncio tasks (_sink_task, _video_out_task) are not being fully released or terminated when a connection is closed or the pipeline task ends.
    • These lingering resources consume memory and system resources (like thread handles).
    • When a new connection attempts to create a new pipeline instance, it fails because the accumulated resources prevent initialization (e.g., cannot create new threads or allocate memory).

Attempted Workarounds

  • I tried limiting the size of RawVideoTrack._video_buffer using maxsize and handling the asyncio.QueueFull exception to drop frames. This successfully prevented that specific queue from growing indefinitely (log warnings confirmed frame dropping), but it did not solve the underlying memory leak or reconnect issues in the filesrc scenario, and the application still crashed with MemoryError or GLib errors originating earlier in the pipeline or during reconnect.
  • Suggestions included also limiting BaseOutputTransport._video_out_queue and configuring the GStreamer appsink element with max-buffers and drop=True to improve backpressure, but the core issue now appears to be related to resource lifecycle management.
  • I am planning to use tracemalloc to get definitive proof of which objects are leaking and where they are allocated.

Expected behavior

  • For high-rate sources, memory usage should stabilize, potentially with frames being dropped if backpressure mechanisms (limited queues, appsink dropping) are implemented correctly. No OOM errors should occur.
  • For finite sources, memory usage should decrease back towards baseline after the source finishes and frames are processed/sent.
  • Upon disconnection, all pipeline-related resources (GStreamer, aiortc, Pipecat tasks/objects) should be cleanly released.
  • Memory usage should return to near baseline after disconnection and garbage collection.
  • Subsequent connections should function identically to the first connection without resource exhaustion errors.

ken-kuro avatar May 02 '25 07:05 ken-kuro