Browsers warn "VideoFrame was garbage collected without being closed"

[jprochazk]

When playing back video, the console periodically receives this error message:

localhost/:1 A VideoFrame was garbage collected without being closed. Applications should call close() on frames when done with them to prevent stalls.

The thing is... we are calling close on every frame already:

https://github.com/rerun-io/rerun/blob/5df2aed73f64fdcd1e6b6609a94d8064262b4c29/crates/viewer/re_renderer/src/renderer/video/decoder/web.rs#L25-L29

The instant we receive a web_sys::VideoFrame from the decoder, we wrap it in a struct that calls close on drop:

https://github.com/rerun-io/rerun/blob/5df2aed73f64fdcd1e6b6609a94d8064262b4c29/crates/viewer/re_renderer/src/renderer/video/decoder/web.rs#L89-L97

I believe it should be impossible for us to not be calling close.

In the first place, the browser VideoDecoder refuses to decode more than ~10 frames at a time, and until the oldest one is closed, it will not produce any more, no matter how many EncodedVideoChunks we give it. That's the "stall" the error message is talking about. Our decoder would not work at all if we weren't calling close! Either I'm missing something obvious, or there's a bug in Chromium.

This doesn't seem to cause any issues such as memory leaks or unnecessary slowness of any kind, so it's only an annoyance, because we would like to not be spewing errors into the console all the time.

[emilk]

I did a quick test:

--- a/crates/viewer/re_renderer/src/video/decoder/web.rs
+++ b/crates/viewer/re_renderer/src/video/decoder/web.rs
@@ -23,6 +23,7 @@ struct VideoFrame(web_sys::VideoFrame);
 
 impl Drop for VideoFrame {
     fn drop(&mut self) {
+        re_log::debug!("Dropping frame at {:?}", self.0.timestamp());
         self.0.close();
     }
 }
@@ -90,6 +91,7 @@ impl VideoDecoder {
         let decoder = init_video_decoder({
             let frames = frames.clone();
             move |frame: web_sys::VideoFrame| {
+                re_log::debug!("Received a frame at {:?}", frame.timestamp());
                 frames.lock().push((
                     TimeMs::new(frame.timestamp().unwrap_or(0.0)),
                     VideoFrame(frame),
@@ -189,10 +191,14 @@ impl VideoDecoder {
             return FrameDecodingResult::Pending(self.texture.clone());
         };
 
+        re_log::debug!("{} frames loaded before drain", frames.len());
+
         // drain up-to (but not including) the frame idx, clearing out any frames
         // before it. this lets the video decoder output more frames.
         drop(frames.drain(0..frame_idx));
 
+        re_log::debug!("{} frames loaded after drain", frames.len());
+
         // after draining all old frames, the next frame will be at index 0
         let frame_idx = 0;
         let (_, frame) = &frames[frame_idx];
@@ -222,6 +228,7 @@ impl VideoDecoder {
     ///
     /// Does nothing if the index is out of bounds.
     fn enqueue_segment(&self, segment_idx: usize) {
+        re_log::debug!("enqueue_segment {segment_idx}");
         let Some(segment) = self.data.segments.get(segment_idx) else {
             return;
         };
@@ -243,6 +250,7 @@ impl VideoDecoder {
         } else {
             EncodedVideoChunkType::Delta
         };
+        re_log::debug!("enqueue_sample at {} ms", sample.timestamp.as_ms_f64());
         let chunk = EncodedVideoChunkInit::new(&data, sample.timestamp.as_ms_f64(), type_);
         chunk.set_duration(sample.duration.as_ms_f64());
         let Some(chunk) = EncodedVideoChunk::new(&chunk)

It starts off enquing the whole video:

Then frames starts coming in, and pretty soon there are over a thousand live VideoFrame:s at the same time.

That's a lot.

I think the issue is that we always pre-fetch two segments, and each segment consists of many samples = many frames.

[emilk]

In the first place, the browser VideoDecoder refuses to decode more than ~10 frames at a time, and until the oldest one is closed, it will not produce any more, no matter how many EncodedVideoChunks we give it.

That's not how my browser acts (Brave/chromium). Brave just keeps decoding everything we ask it to decode, leading to having thousands of decoded frames in memory at the same time, eating up RAM.

[jprochazk]

Then frames starts coming in, and pretty soon there are over a thousand live VideoFrame:s at the same time.

This should not cause the browser to complain about video frames being garbage collected before being closed. I mean, we're holding onto them, so they're not being garbage collected. And once we aren't using them, even if we have thousands of them, they should all still be closed.

I think the issue is that we always pre-fetch two segments, and each segment consists of many samples = many frames.

That's not how my browser acts (Brave/chromium). Brave just keeps decoding everything we ask it to decode, leading to having thousands of decoded frames in memory at the same time, eating up RAM.

How much RAM is it eating up? If it's an unacceptable amount then we'll have to deal with this somehow. But it should be opened as a separate issue

[emilk]

Let's see if we can do a minimal repro and report it upstream to chromium

[Wumpf]

Does no longer repro with Chrome 137.0.7151.69 on Windows Want to try on Mac as well

[Wumpf]

Also doesn't happen on Chrome 137.0.7151.70 on Mac either anymore. Looks like this got fixed!