Video via RTC not reliably working (workaround available)

[mshopf]

Version

• Patreon name: Matthias Hopf
• Phaser Version: 3.55.2
• Operating system: Ubuntu 20.04.4 LTS
• Browser: Chrome 102.0.5005.61, Vivaldi 5.2.2623.48

Description

Opening a RTC video stream in an phaser video object is not working always; in particular, streaming a screen share seems to work, streaming a webcam doesn't. Opening the according streams on the local computer always works. I tested this with 2 different web cams on linux, on windows and on android, consistent behavior. I have the RTC remote stream always working when creating the video DOM object myself, so it's probably a minor issue to fix, however, I'm lost in phaser's code regarding this (I guess this has even something to do with the webgl renderer).

Unfortunately, a fully working example is - as typical with RTC - about 650 LoC, plus the signaling server. I can setup one in the near future if required for reproduction, but due to a cyber attack at our university, the currently running server is not accessible from the outside.

Example Test Code

This is how I setup the phaser video object; v is the storage object for a single video element. The update function is used to scale the video to a maximum of 200x200, and indicates that the video is actually available. For remote WebCam RTC streams, video size remains at 2x2, and nothing is displayed. For all other video variants, video size is updated after a few ticks, and the video is played (video is displayed a little earlier than the update function is called, thus too large on screen for a few frames).

v.phaser_vid  = mainScene.add.video ();
v.phaser_cont = mainScene.add.container (Math.random()*660, Math.random()*300, v.phaser_vid);
v.phaser_cont.setSize (140, 140);
mainScene.physics.world.enable (v.phaser_cont);
v.phaser_cont.body.setVelocity (10 + Math.random() * 200, Math.random() * 100)
.setBounce (1, 1) .setCollideWorldBounds (true);
    // Following 2 lines are to be replaced by the code in the next snippet for a workaround
    v.phaser_vid.loadMediaStream (stream);
    v.phaser_vid.play();
v.phaser_vid.video.addEventListener ('timeupdate', updateSize);
function updateSize () {
    if (v.phaser_vid.width > 3 && v.phaser_vid.height > 3) {
        console.log ('updateSize '+v.phaser_vid.width+'x'+v.phaser_vid.height);
        var width  = v.phaser_vid.width / v.phaser_vid.height * 200;
        var height = 200;
        if (width > 200) {
            width = 200;
            height = v.phaser_vid.height / v.phaser_vid.width * 200;
        }
        v.phaser_vid.setDisplaySize (width, height);
        v.phaser_vid.video.removeEventListener ('timeupdate', updateSize);
    }
}

If I setup the video object the following way (instead of using .loadMediaStream() and .play()), everything works as intended. Interestingly enough, I don't need to send any signals to phaser, it automatically detects the video size change, indicating that the video truely wasn't setup correctly.

v.video_ready = false;
v.dom_vid = dom_setup_video (stream, muted, function () { v.video_ready = true; });
v.phaser_vid.video = v.dom_vid;             //instead of .loadMediaStream() and .play()
function dom_setup_video (src, muted, cb) {
    const video    = document.createElement ('video');
    var playing    = false;
    var timeupdate = false;
    // Wait for video frames to be accessible
    video.addEventListener ('playing', playingCB, true);
    function playingCB () {
        playing = true;
        checkReady();
    }
    video.addEventListener ('timeupdate', timeupdateCB, true);
    function timeupdateCB () {
        timeupdate = true;
        checkReady();
    }
    video.autoplay  = true;
    video.muted     = true;
    video.loop      = true;
    video.srcObject = src;
    video.play();
    return video;
    function checkReady() {
        if (playing && timeupdate) {
            cb();
            video.removeEventListener ('playing',    playingCB,    true);
            video.removeEventListener ('timeupdate', timeupdateCB, true);
        }
    }
}

Additional Information