OES Rendering Path on Oculus Quest is rendering in a Sphere below the Camera instead of around the Camera

[LB-Till]

Issue description When trying to use OES Path with ApplyToMesh on Oculus Quest, the video is rendering on a small sphere below the camera instead of a sphere around the player.

Also when choosing Multiview, the one eye gets an offset.

My Setup

• Unity version: 2021.3.0f1 (LTS)
• AVPro Video version (number and edition (trial/core/ultra/enterprise)): AVPro Video Android Core 2.5.2
• Operating system version: Android - Oculus Quest (41.0.0.xxx)
• Device model: Oculus Quest
• Video specs (resolution, frame-rate, codec, file size): Cones-360Mono-4K30-H264 from Demo Videos
• URP 12.1.6
• XR Plugin Management 4.2.1
• Oculus XR Plugin 3.0.2

To Reproduce

1. Create an empty project,
2. Import XR Plugin Management 4.2.1
3. Import Oculus XR Plugin 3.0.2
4. Import Universal Render Pipeline 12.1.6
5. Create and Add URP Pipeline Asset under Graphic Settings
6. import AVPro Video Core Android Edition 2.5.2
7. Set in Project Settings: 7a. Graphics API to "Open GLES3" 7b. Color Space Linear
8. Create new Scene, add Tracked Pose Driver to Main Camera
9. Add AVPro Video - Media Player 9a. Add Video 9b. In Platform Specific Settings - Android: Set Use OES Rendering to enabled
10. Copy "Sphere-ApplyToMesh" from Demo_360Stereo scene. (Should have 100,100,100 Scaling and Pos at 0,0,0) 10a. Fill in Media Player 10b. Swap Shader for the Demo-360 Material to InsideSphere Unlit - Android OES ONLY
11. Build Scene and Run.

Logs

Screenshots Screenshots are ODH captures, so ignore the double eye effect.

Screenshot with multiview

Videos

[Ste-RH]

I only have a Quest 2 to hand - which I wouldn't think should make a difference.

Following your steps exactly, I do not get what you are seeing. I get a perfectly rendered sphere around the camera playing back the video as expected. This is in both Multi Pass and Multiview.

Can you email over the project you created so I can see what might be different? If so, send it to [email protected]

[LB-Till]

Ok, just confirmed that i do have the same issue on oculus quest 2.

At this point, I assume i forgot a checkbox somewhere or used a weird setting. I've sent the project to you guys. Thanks in advance.

[Ste-RH]

Thanks for testing/confirming re Quest 2, and sending over a project. Have requested access to the download.

[Ste-RH]

The difference is quote simple.

You are not using the Mono sample video, you are using the Stereo sample video. This is a top/bottom stereo packed. If I change it to the Mono video, everything is rendered fine. If I leave it as the stereo video and change the packing to top/bottom, everything is fine.

[LB-Till]

ok. I've just tested it. Set stereo packing to TopBottom, kept stereo video. Same Result as before: With sphere at bottom and double image when using multiview. Also tried keeping stereo at none and taking the mono video. Same result.

[LB-Till]

Is there a shader to use the OES path on the skybox instead of the sphere? Might be an option if that is working.

[Ste-RH]

Multi Pass, Multiview, both OES and none-OES all behave exactly as they should do here.

The only other thing I did was zero the camera transform position...but that shouldn't do anything.

No, there is no skybox OES shader. Only the special AVPro Video ones.

[Ste-RH]

[Ste-RH]

Anything to report @LB-Till ?

[LB-Till]

Sorry for the late reply. We've had a bank holiday and a long weekend over here last thursday.

I've just created a new build. Reset camera and checked all the settings. Issue persists.

[LB-Till]

Ok, i think i found a quick fix. There seems to be an issue with the conversion of the model itself. I created a sphere in blender and it was tiny in the build. Once i increased the scale factor in the import settings, it has the increased size. So i assume, for some reason my import is not working correctly.

With the custom sphere i do have some clipping with the corners though, which is probably because my sphere isn't as smooth as yours?

EDIT: Multiview is still broken for me, rendering two separate images that overlap. (Only the rendering sphere though, a default cube inside the sphere is rendering fine)

[Chris-RH]

Yeah, in blender you need to make sure you've used the modifier "subdivision surface" and increase the levels. Also Object menu -> shade smooth.

Is it just your own sphere that is not rendering properly still or is it the default demoscene one?

[LB-Till]

Neither sphere is rendering correct in multiview (separate images for each eye). Multiview itself is working correctly though. Default sphere is still rendering too small. My own sphere is rendering fine, except for the smoothing issue (which can be probably solved easily, by increasing the polycount).

I tried all different combinations of shader and media player settings. And using both mono and stereo videos. To me it looks like an issue in the shader or shader execution, but i can't get my head around it. What or why its not working properly.

In the screenshots below you can see the difference between left and right eye. Both screenshots using the custom sphere. The cube is fine for both eyes, the video is separated. The yellow spot is a near plane clipping issue, that requires some finetuning.

[Ste-RH]

Can you email over another project showing the above so we can take a look?

[LB-Till]

sure, but its still the same barebone project, just swapped out the video and the sphere.

[LB-Till]

sent it. Im really curious if you can reproduce it or not. Because if not, there must be something incredibly stupid im missing.

[Ste-RH]

Thanks for sending over the project. I just want to make sure we are looking at the same baseline.

Building/running without any changes at all on a Quest 2...and grabbing a video via Android Studio, this is what I get:

https://user-images.githubusercontent.com/46531149/175299205-3bcb1309-e8c5-49c6-bb56-17116769d874.mp4

[LB-Till]

that looks pefectly fine to me. No shifted image. Just the near plane glitch, which we can ignore. what the hell is going on in my project

[Ste-RH]

that looks perfectly fine to me. No shifted image.

That is what I thought

[LB-Till]

ok, i will do a completely new setup on a different computer. And see if i get the same issue there.

[LB-Till]

Tested on a different computer: Importing the project i sent you & building without changing anything: Exactly same issues as before Setting up a clean & new project fixed the "tiny sphere" issue with the demo sphere. But im still having the shifted image issue. So at least now i can confirm, that the sphere scale and the multiview issue were separate issues.
I also tested the apks on a different device (quest 2) to rule out its a device issue.

I will continue to investigate the issue, but im absolutely confused whats happening here.

[LB-Till]

My findings so far: Using the URP Pipeline Asset in Graphic Settings leads to the "tiny sphere". Removing it, displays the sphere in the regular (working) size.

[LB-Till]

I recently tested this on my home computer with a complete fresh unity setup (2021.3.5f1 LTS). Fresh project, only downloading urp, xr plugin management, oculus plugin and avpro video core android.

I created a URP Pipeline Asset and added it to Graphics and Quality in the Settings. I disabled Auto Graphics API and kicked out Vulkan and then switched to linear color space. In XR Plugin Management i enabled Oculus and in the Oculus Settings i switched to multiview and turned everything off except for target devices.

Using the Demo 360 Stereo Scene:

• in the media player swapped out the Reference to the mono video
• added a tracked pose driver to the main camera
• removed the LookAround360 and Update Multi-Pass Stereo scripts from the main camera
• (swapped out the shader on ApplyToMesh to Inside Sphere Unlit Android OES Only)
• (enabled Use OES Rendering)

It works absolutely fine until i enable oes rendering and use the OES shader. Then the sphere gets rendered scaled down and distorted image in the right eye.

I am completely at my wits end here.

[Ste-RH]

I followed your instructions almost to the letter. The only difference was I made a new URP Global Settings before I made a URP Asset (see screen shot). I get the following output.

https://user-images.githubusercontent.com/46531149/176488588-278fbeeb-b45d-455b-8b44-35418f24df10.mp4

Looks exactly as I would expect (yes, this is OES).

[LB-Till]

thanks, i'll give it another try.

[LB-Till]

unfortunately its still the same. Im pretty sure the URP Global Settings gets generated automatically when assigning the URP Asset to Graphics though.

Could there be an issue during the build process maybe? Are you using the Android SDK shipped with Unity? Or could it be some sort of issue with a unity package that may gets autoloaded for my projects? This all makes absolutely no sense 😕

Also what i've noted, that after building, unity has done some changes with the x86 .so files in the android folder even though ARM64 is selected as target architecture. I have no idea if that is normal, i just found it odd.

[Ste-RH]

Are you using the Android SDK shipped with Unity?

I used a complete stock fresh install of the Unity version you said you were using.

The x86 is metafile differences. Interesting as I would actually expect to see no changes, or more than just x86. That is something I will look into...but it will be nothing to do with the odd things you are seeing. You could delete the x86 and x86_64 folders and it will (should!) have no impact/difference.

[Ste-RH]

Your architecture screen grab is interesting. Are you building with IL2CPP? By default Unity sets the Mono scripting backend. Only by changing it to IL2CPP can I then set ARM64. As you never stated a scripting backend change I have always left it on Mono. Will try IL2CPP now...

[Ste-RH]

Again, exactly as it should be.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.