WebRTCStereoVideoPlane can no longer split the image

[wy-chien]

Hi, thank you for constructing this amazing repo.

Currently, I am working on a VR project(Open Television) that can pass a horizontally stacked stereo pair image to the frontend to preview the stereo image. The camera is zed mini, PC is Ubuntu 22.04 x86_64, Quest 3 via reverse TCP.

However, for veur==0.0.32rc7, it works fine in the web view. However, after entering VR mode, the headset's orientation becomes weirdly horizontal so that you will see the ground grid stay vertically.

web view:

Wrong ground or camera orientation in VR

I then upgraded the vuer to some later versions. But things get even weirder for veur==0.0.46+, the WebRTCStereoVideoPlane can no long split the horizontally stacked image but squashes each of the images to be half-narrower.

The image plane in the VR is no longer a stereo view but just a hstack image with left and right images. This happens in both web view and VR mode, but the image plane in the web view keeps flashing and VR mode does not.

Here's the part that Open Television calls this function

async def main_webrtc(self, session, fps=60):
        session.set @ DefaultScene(frameloop="always")
        session.upsert @ Hands(fps=fps, stream=True, key="hands", showLeft=False, showRight=True)
        
        # Get local network info
        host = self.app.host
        # Always use HTTP for WebRTC to avoid SSL handshake issues
        protocol = "http"
        
        # Get client's IP address from the session if possible
        try:
            client_ip = session._raw_session.request.remote
            print(f"Client connected from: {client_ip}")
        except Exception as e:
            client_ip = None
            print(f"Could not determine client IP: {e}")
        
        # Determine the WebRTC URL based on network setup
        webrtc_port = 8080
        server_ip = "localhost"
        
        # Try to get the local IP address that's reachable from the network
        try:
            # Create a socket and connect to a public server
            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
            s.connect(("8.8.8.8", 80))
            server_ip = s.getsockname()[0]
            s.close()
        except Exception as e:
            print(f"Could not determine local IP: {e}")
        
        # Determine the correct WebRTC URL
        if hasattr(self.app, 'ngrok_url') and self.app.ngrok_url:
            # Use ngrok URL if available (priority)
            webrtc_base = self.app.ngrok_url.rstrip('/')
            webrtc_url = f"{webrtc_base}/offer"
            print(f"Using ngrok WebRTC URL: {webrtc_url}")
        else:
            # Use local network URL
            webrtc_url = f"{protocol}://{server_ip}:{webrtc_port}/offer"
            print(f"Using local WebRTC URL: {webrtc_url}")
        
        print(f"\nTo connect to the WebRTC stream directly in a browser, visit:")
        print(f"{protocol}://{server_ip}:{webrtc_port}\n")
        
        # Create the stereo video plane
        session.upsert @ WebRTCStereoVideoPlane(
            src=webrtc_url,
            key="zed",
            aspect=1.66667,
            height=10,
            position=[0, -2, -0.2],
        )
        
        while True:
            await asyncio.sleep(1)

Do you have any clue where this issue is from, or did I miss passing any argument for later versions?

[geyang]

Thanks for filing this comprehensive issue, I recently build a stand-alone example repo for teleoperation and stereo streaming here: https://github.com/vuer-ai/webrtc-teleop-demo

There are a few updates that I needed to add that is in a private repo. I will try to patch it up tomorrow.

[geyang]

I just updated the repo, it should be up-to-date now!

[wy-chien]

Thank you for your quick response. I am trying to run the repo with the command python 01_stereo_video.py, but I get an empty scene. Do you have any instructions to run with Zed Mini?

Plus, how can I eliminate the red lines that overlay the video plane?

[geyang]

use 02 and 03. 01 hasn't been updated yet.

[jizhaox]

The image plane in the VR is no longer a stereo view but just a hstack image with left and right images. This happens in both web view and VR mode, but the image plane in the web view keeps flashing and VR mode does not.

@wy-chien Hi, I am using the verion 0.0.60, and have the same issue as above. Which version works for you, and how did you solve it? Thanks!

[geyang]

@marvinluo1 This one needs to be attended to

[wy-chien]

The image plane in the VR is no longer a stereo view but just a hstack image with left and right images. This happens in both web view and VR mode, but the image plane in the web view keeps flashing and VR mode does not.

@wy-chien Hi, I am using the verion 0.0.60, and have the same issue as above. Which version works for you, and how did you solve it? Thanks!

I was using 0.0.46 to resolve the issue for now. I am willing to upgrade to any new version if you have tried anything that works.

[23369]

I also have met the same qeustion, vuer version=0.0.61, The video is rendered as a single side-by-side panorama; left and right eyes see the full frame, which cannot come true the stereo videos.