Memory leak using ImageRecorder widget from ipywebrtc

[dcabecinhas]

I'm trying to create a high quality movie out of a pythreejs scene animation by saving sequential images using ImageRecorder, as described in the examples.

However, I noticed there is a memory leak that leads Chrome to occupy all my RAM after a few iterations. I'm attaching a minimal example using a texture to drive memory occupation faster.

I cannot pinpoint exactly where the problem is as I'm not knowledgeable of pythreejs or Jupyterlab internals.

To replicate: Run the code and click the 'snapshot' button repeatedly (10x, 100x)

Expected outcome: Minimal change to Chrome memory usage

Current outcome: Chrome memory usage skyrockets (as seen in Activity monitor)

OS: MacOS 10.15.4 Pythreejs: 2.2.0 ipywebrtc: 0.5.0

from pythreejs import *
import numpy as np
from IPython.display import display
from ipywidgets import HTML, Output, VBox, HBox, jslink

#
# Create checkerboard pattern
#

# tex dims need to be power of two.
arr_w = 2048
arr_h = 2048

import numpy as np

def gen_checkers(width, height, n_checkers_x, n_checkers_y):
    array = np.ones((width, height, 3), dtype='float32')
    
    # width in texels of each checker
    checker_w = width / n_checkers_x
    checker_h = height / n_checkers_y
    

    for y in range(arr_h):
        for x in range(arr_w):
            color_key = int(x / checker_w) + int(y / checker_h)
            if color_key % 2 == 0:
                array[x, y, :] = [ 0, 0, 0 ]
            else:
                array[x, y, :] = [ 1, 1, 1 ]
    return array
                

data_tex = DataTexture(
    data=gen_checkers(arr_w, arr_h, 4, 4),
    format="RGBFormat",
    type="FloatType",
)

ball = Mesh(geometry=SphereGeometry(radius=1, widthSegments=32, heightSegments=24), 
            material=MeshStandardMaterial(color='white', map=data_tex),
            position=[1, 0, 0])

c = PerspectiveCamera(position=[0, 5, 5], up=[0, 1, 0],
                      children=[DirectionalLight(color='white', position=[3, 5, 1], intensity=0.5)])

scene = Scene(children=[ball, c, AmbientLight(color='#777777')])

renderer = Renderer(camera=c, 
                    scene=scene, 
                    controls=[OrbitControls(controlling=c)],
                    antialiasing=True
                   )
display(renderer)

import ipywebrtc
from time import sleep
stream = ipywebrtc.WidgetStream(widget=renderer)

image_recorder = ipywebrtc.ImageRecorder(filename=f"snapshot", format='png', stream=stream)

out = Output()
VBox([image_recorder, out])

[vidartf]

I cannot seem to replicate this. Are you using classic notebook, or JupyterLab? (it should hopefully not matter, but best to try as close to your conditions as possible).

[vidartf]

Also: which version of chrome? If not latest, can you try upgrading it?

[dcabecinhas]

Thanks for taking your time to look at this. I have the most recent Chrome (Version 81.0.4044.138 (Official Build) (64-bit)) and I'm running Jupyterlab 1.2.6.

I tuned the example a bit (larger canvas, texture, and automatic snapshot) so the leak is more noticeable. Now, clicking the "snapshot" once takes n snapshots in a loop . In my computer (with other tabs open) I get

After initial rendering, before clicking "snapshot":

Google Chrome Helper (GPU): 2.47 Gb
Google Chrome Helper (Renderer): 669.3 Mb

After 50 snapshots:

Google Chrome Helper (GPU): 3.68 Gb
Google Chrome Helper (Renderer): 1.21 Gb

Example code:

from pythreejs import *
import numpy as np
from IPython.display import display
from ipywidgets import HTML, Output, VBox, HBox, jslink

# Create texture pattern

tex_size=4*1024

bound=10*np.pi
x = np.arange(-bound/2,bound/2,bound/tex_size,dtype=np.float32).reshape(-1,1)
y = np.arange(-bound/2,bound/2,bound/tex_size,dtype=np.float32).reshape(1,-1)

pattern = np.abs(np.cos(x)*np.sin(y))
pattern = np.stack([pattern,pattern,pattern],axis=-1)

data_tex = DataTexture(
    data=pattern,
    width=tex_size,
    height=tex_size,
    format="RGBFormat",
    type="FloatType",
)

ball = Mesh(geometry=SphereGeometry(radius=1, widthSegments=16, heightSegments=8), 
            material=MeshStandardMaterial(color='white',map=data_tex))

c = PerspectiveCamera(position=[3, 0, 3],
                      children=[DirectionalLight(color='gray', position=[3, 5, 1], intensity=0.5)])

scene = Scene(children=[ball, c, AmbientLight(color='#777777')])

renderer = Renderer(camera=c, 
                    scene=scene,
                    height=600,
                    width=600,
                    controls=[OrbitControls(controlling=c)],
                    antialiasing=True
                   )
display(renderer)

import ipywebrtc
stream = ipywebrtc.WidgetStream(widget=renderer)

image_recorder = ipywebrtc.ImageRecorder(filename=f"snapshot", format='png', stream=stream)

out = Output()

numImage = 0
numImages = 100

def savePicture(_):
    with out:
        global numImage
        global numImages
        if numImage >= numImages:
            return

        if( numImage % 5 == 0):
            out.append_stdout("Image " + str(numImage) + " of " + str(numImages) + " captured \n")
        renderer.render(scene,c)        
        numImage = numImage + 1        
        ball.position = (numImage/100,0,0)        
        image_recorder.recording = True

image_recorder.image.observe(savePicture, names=['value'])   
VBox([out,image_recorder])

[dcabecinhas]

Other datapoint, that might (or not) be related:

In Safari 13.1 by hand-clicking the snapshot button I see memory go up but eventually gets released after a short time (few seconds). However, I am unable to get the "automated recording loop" working. It just runs the savePicture() function once. It is like the image_recorder.recording = True inside savePicture() doesn't register.

Using a video as stream for the ImageRecorder, the same code works to create a "record image" loop.