CPU-heavy task causes frame delay

[Kogia-sima]

trafficstars

Summary

CPU-heavy task inside the while-loop causes video frame delay within the streamlit-webrtc container.

minimal reproduction:

import time
from streamlit_webrtc import webrtc_streamer


ctx = webrtc_streamer(key="example")

while ctx.state.playing:
    # busy loop
    start = time.time()
    while time.time() - start < 0.1:
        pass

Details

I'm building a real-time image recognition application using streamlit-webrtc. However, the video in the application is extremely laggy and delayed. I examined this and found that the above minimal code causes same behavior. I also found that this issue has gone after I inserted manual sleep inside the loop. However, I can't figure out how long sleep is enough to avoid frame delay issue.

ctx = webrtc_streamer(key="example")

while ctx.state.playing:
    # busy loop
    start = time.time()
    while time.time() - start < 0.1:
        pass

    # manual sleep
    time.sleep(0.05)

Question

• Is there any way to avoid lag issue without a manual sleep?
• If not, is there smarter way to wait until the application draws the latest video frame?

Environment

Context	Version
OS	Windows 10 Pro 21H2 (19044.3930)
Shell	PowerShell 5.1.19041.3930
Browser	Google Chrome 121.0.6167.140 (stable)
Python	3.11.5
streamlit	1.30.0
streamlit-webrtc	0.47.1

[thegenerativegeneration]

I have the same problem, although I'm not even doing anything particularly heavy in my loop.

while ctx.state.playing:
    with lock:
        is_speaking = info["is_speaking"]
        audio_frame_id = info["audio_frame_id"]

I even get delay if the only thing in the while loop is:

while ctx.state.playing:
    pass

The lines above cause a delay of multiple seconds in the video delivered via webrtc:

def video_frame_callback(frame): return frame

My environment is:

OS: macOS Ventura 13.4.1 (arm) Shell: zsh Browser: Firefox 122 Python: 3.10.13 streamlit: 1.31.1 streamlit-webrtc: 0.47.1

[whitphx]

I think it's because Python doesn't execute CPU-bound tasks in parallel even in multiple threads due to the GIL. Off-loading the cpu-heavy tasks to another "process" by using multiprocessing might be a solution.