Streaming video update

[nikky4D]

Is your feature request related to a problem? Please describe.
I would like a Gradio equivalent to cv2.imshow() to process the frames of a video, and see the update in realtime/streaming fashion.

Describe the solution you'd like
When processing video, we use cv2.VideoReader to pull a frame, make modifications to that frame, and display the modified frame with cv2.imshow(). I would like something like this in Gradio.

Additional context
Currently for a video demo, once the user has changed the inputs in some fashion, to see the results, the following have to happen: 1) the model has to process the frames of the video, 2) the modified frames are then compiled back into a video, 3) this video is saved somewhere and displayed. I would like to avoid having to do this intermediate saving and just show each modified frame with a gradio equivalent to cv2.imshow for streaming frames.

[JefferyChiang]

#1475

[abidlabs]

Thanks @nikky4D, @JefferyChiang. Also tagging @tomriddle54 who opened the related issue. We're looking into this!

[aisensiy]

Is any one working on this feature?

[abidlabs]

@aliabid94 is this something you can take a look into as part of the 3.2 milestone?

[NotAnyMike]

I haven't used Gradio, this feature is the main one missing for me to start using it, I wanted to know if I can use rtsp streaming for inference? is there any workaround for this? or the only way is to wait for this feature?

[1lint]

I was working on something similar and found you can stream a processing video by yielding the result each time you process a video frame. Here's a basic example

import gradio as gr
import cv2

def process_video(input_video):
    cap = cv2.VideoCapture(input_video)

    output_path = "output.mp4"

    fps = int(cap.get(cv2.CAP_PROP_FPS))
    width  = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    video = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (width, height))

    iterating, frame = cap.read()
    while iterating:

        # flip frame vertically
        frame = cv2.flip(frame, 0)
        display_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        video.write(frame)
        yield display_frame, None

        iterating, frame = cap.read()

    video.release()
    yield display_frame, output_path

with gr.Blocks() as demo:
    with gr.Row():
        input_video = gr.Video(label="input")
        processed_frames = gr.Image(label="last frame")
        output_video = gr.Video(label="output")

    with gr.Row():
        examples = gr.Examples(["parrot.mp4"], inputs=input_video)
        process_video_btn = gr.Button("process video")

    process_video_btn.click(process_video, input_video, [processed_frames, output_video])

demo.queue()
demo.launch()

It works well when running locally, I also have the example deployed on hf spaces https://lint-realtime-video.hf.space/ but it processes real slow/freezes up sometimes

[abidlabs]

Hi @nikky4D and all, sorry for the late response, but this is doable through Gradio's support of iterative outputs. As @1lint points out, your function can simply yield a sequence of images to render them in real time.

Here's an example demo that you may find useful: https://github.com/gradio-app/gradio/blob/main/demo/fake_diffusion/run.py

Please let me know if you have a different use case that this doesn't cover @nikky4D.

[tonydavis629]

I am finding that the images returned are quite slow, at maybe 2 or 3 frames per second. Where should I be looking to speed this up? My generator is operating at 300 FPS, and I'd like to see the images update at 30 FPS. Is there any way to control this with gradio?

[freddyaboulton]

Hi @tonydavis629 - gradio does some post-processing to the images (convert to base64) to send back to the client. Maybe that's slowing you down. If you file a new issue with reproducible codewe can take a look. I suspect we'll have to make changes to gradio to support near real time communication.

[tonydavis629]

@freddyaboulton can you point me to exactly where this post processing is done? I could potentially submit a PR to short that circuit if desired

[freddyaboulton]

The image has to be converted to base64 cause we return JSON to the client. I think the better approach is to use the streaming response introduced in #5077

[1lint]

I am finding that the images returned are quite slow, at maybe 2 or 3 frames per second. Where should I be looking to speed this up? My generator is operating at 300 FPS, and I'd like to see the images update at 30 FPS. Is there any way to control this with gradio?

You could yield an image every 10 processed frames in that case, should go much faster.