Unreliable capture of pythreejs with ImageRecorder on Safari

[jpanetta]

I am trying to capture an image of a pythreejs renderer using ImageRecorder. This works reliably on Chrome but not on Safari. I seem to be hitting a weird race condition, but I do not understand the asynchronous Javascript code involved well enough to figure what is going wrong.

Here is an example demonstrating the problem:

from pythreejs import *
ball = Mesh(geometry=SphereGeometry(),
            material=MeshLambertMaterial(color='red'))
light    = DirectionalLight(color='white', position=[3, 5, 1])
c        = PerspectiveCamera(position=[0, 5, 5], children=[light])
scene    = Scene(children=[ball, c])
renderer = Renderer(camera=c, scene=scene, controls=[OrbitControls(controlling=c)])
display(renderer)
#-----------------------------------------------
import ipywebrtc
stream = ipywebrtc.WidgetStream(widget=renderer, max_fps=30)

rec = ipywebrtc.ImageRecorder(filename='capture', format='png', stream=stream)
rec.autosave = True
#-----------------------------------------------
rec.recording = True
#-----------------------------------------------
rec.recording
# Usually still prints `True` :(

(Comments delimit notebook cells.)

I expect rec.recording = True to (a) rerender the scene, (b) capture and save an image, and (c) reset rec.recording to False. This all happens reliably on Chrome, but (b) and (c) almost always fail on Safari (no image is written, and rec.recording is still True at the end). Manually triggering a second redraw (e.g., by orbiting the view) usually causes (b) and (c) to finally happen.

As best I can tell, ImageRecorder's snapshot function hangs on the onCanPlay call. It appears the redraw requested by pythreejs here is somehow happening "too soon," but I don't understand what order everything needs to happen in.

This is tricky to debug since the capture starts working more (but not 100%) reliably when I insert breakpoints, e.g. here and here.

Do you have any ideas what might be wrong or suggestions on how to debug this?

[jpanetta]

I can actually reproduce this problem with just an IntSlider, so it seems the issue is not specific to pythreejs after all:

import ipywebrtc, ipywidgets

slider = ipywidgets.IntSlider()
slider
#------------------------------------
class ScreenshotWriter():
    def __init__(self, widget):
        stream = ipywebrtc.WidgetStream(widget=widget)
        self.rec = ipywebrtc.ImageRecorder(format='png', stream=stream)
        self._i = 0
        self.rec.image.observe(self._imageUpdated, names='value')
    
    def _imageUpdated(self, change):
        if (self.rec.image.value):
            self.rec.save(f'screenshot{self._i}.png')
            self._i += 1
            self.rec.image.value = b''
    
    def capture(self):
        self.rec.recording = True

w = ScreenshotWriter(slider)
#------------------------------------
w.capture()
#------------------------------------
w.rec.recording # after running the above cell a few times, this gets stuck at `True`

Capture seems slightly more reliable for the slider, but after writing a few images, eventually rec.recording gets stuck at True and images stop getting written. Chrome still works reliably.

[jpanetta]

Strangely after updating to Safari 14.0 (from 13.1.2), now the IntSlider example works reliably, but the pythreejs example still is still flaky: the recorder will get stuck and I have to force the renderer to redraw to unstick it.