Scaling the mesh down moves it farther away

[parkchamchi]

As mentioned here and here.

[parkchamchi]

Turns out that, contrary to what I've believed, lowering ScaleR does not make it go farther from the camera. Consider this dummy depth map, where white part is the closest, This is what it looks like when I put an independent plane at the same position with the mesh, where z-fighting can be observed. And this is what it looks like when I set ScaleR to 0.1. The actual distance is not changed, as the z-fighting demonstrates.

Conclusion: the mesh does not move when ScaleR is changed. It's that when scale is too small it makes an optical illusion -- or it can be said: when the scale is high, as the default setting of this program, makes an optical illusion rendering the user to perceive that the object is closer than it actually is.

Side note: setting DepthMult too high actually pushes the mesh, seems like a floating point precision issue.

[311-code]

I was able to solve the problem of objects being far away when you shrink them and scale too large last night by using a program called OpenVRadvanced settings and setting the playspace offset setting, which forces the application to move the VR camera wherever you want in the space. You can also use spacedrag https://www.youtube.com/watch?v=hFSzvnffNww

Yes it does not move with scaleR after my testing. But the object by default is very far away from the camera in Unity it looks (Edit: NM it's the depthplane mesh is 10 meters away and scaled very large) But the scale of the mesh object is so large it is decieving, it's as tall as a small building. if mesh object is at scale of 1 in unity (not sure which one is it) it may need to be scaled down to like 0.1 or 0.2 (I'm not sure which object in Unity is the one that is the mesh) This is also why by default there is minimal positional tracking feeling with 6dof on. so when you scaleR it down in the release build it looks like it's shrinking and getting father away but it's just an illusion, the object is just scaled to a massive size by default like a building size in VR and far away from the VR camera and when you shrink it is has a sort of feeling of getting far away but it's not (not noticeable in 2D screen but very noticeable in VR).

So for now I have been using OVRadvanced settings because I can't get Unity project to build to fix this for you. I first set the depthmultrl to my liking for better 3D, then scaleR to 0

Then I use OpenVRadvanced settings (or ovr spacedrag feature) and in the SteamVR menu I set playspace offset to x=0, y=0, z=145, this forces VR camera to the object so it is right next to me, it's perfect scale to reality and positional tracking is normal and noticeable again because everything is correct scale and 1:1, like native VR now.

I just tried the Beit model on a video and it was slower but pretty amazing how accurate the monoscopic 3D conversion was. It basically felt very similar to Marigold to me but much faster than Marigold, and now in VR in correct scale. I found it to be much more immersive than regular VR sbs videos because the the 6dof positional tracking adds all kind of subtle depth cues.

Is there any way to not only have the meshhor, meshver, but an extra meshZ that moves forward and backwards the mesh, so we don't have to use openVRadvanced setting workaround each time, or just adjust the camera in unity? Probably the GUI will have to be shrunk then and both cameras moved to work for both VR and desktop.

[parkchamchi]

But the object by default is very far away from the camera in Unity it looks like (-150) webxr camera location?

No, the WebXRCameraSet was for WebGL build, which was not updated since v0.6.2 (2022 Nov). The actual camera (XR Origin/Camera Offset/MainCamera) is fixed at (0, 0, 0), and the DepthPlane is set by default at (0, 0, 10). IIRC one Unity unit equals one meter thus the initial distance is 10m. This is changed by CamDistL parameter: 10^CamDistLm (Thus the initial CamDistL is 1.

but an extra meshZ

I can, but doesn't the CamDistL do that job? That parameter goes down to -1 btw.

[311-code]

Sorry yes you are correct. I don't know why I estimated 150 meters.. that is way off. When I walk over in VR the to depth plane in the release build it is indeed only about 10 meters away in real life and scaled more large a small bus by default (not a building).

But I did get the project working now and found the best settings for VR for the right scale and distance, the desktop GUI still works also with this setting! The Z being 0.5 being most important as scale can be adjusted after in the scaleR, but its nice to have good scale at default in VR.

Haha, kind of a weird photo here with a disembodied head, but I tested with a coffee cup in mixed reality on the Quest 3 and now the coffee cup feels right behind the image and the correct depth / scale on the head at z 0.5 and scaled down with those settings (or using scaleR), before it would be double image on coffee cup when looking at the large object that was 10m away and hurt my brain. The app having black background for passthrough is sort of an issue for mixed reality, needs a green option for the cutout which I am messing around with now.

[311-code]

for reference here is how big the default scale is when you move DepthPlane closer to the VR camera in Unity. from -10 spot to 0.5 :) So in release build, if we shrink the scale.. it's still staying at 10 and far away in VR, and there is no ability to move it closer because none of the settings allow for moving it in VR on Z axis closer.

My first impressions when seeing this app in VR was that it didn't work right, but it's just small problem that needs to be resolved, could be great for retaining some SteamVR users.

Edit: Maybe this could also all be as simple as editing the mainbehavior or meshbehavior scripts and giving full w, a, s, d movement with the camera, q and e rotate mesh, r and f for up and down tilt, mouse look. ChatGPT4 could implement it in about 3 minutes. This would also enhance the desktop mode (I would get rid of black fade also or make it fade way later.)

[parkchamchi]

The app having black background for passthrough is sort of an issue for mixed reality, needs a green option for the cutout which I am messing around with now.

Use the command set_camera_bg 0 255 0 0 to set the background. (Or put it in the initcmd.txt)

[311-code]

I'm about to implement this https://www.uploadvr.com/quest-3-depth-api-mixed-reality-dynamic-occlusion/ so the coffee cup and my hands will occlude the object. Do you have any plans on adjusting the scale and camera Z location to be correct for VR based on the information I've provided? Or perhaps better control to move in and out in Z direction. The app is actually very difficult to navigate in even desktop mode when it comes to moving in z direction so could use better controls also.

[parkchamchi]

Yeah, I guess I better do it.

I was working on a command that moves the mesh to the controller, and it made the existing parameters somewhat janky. I've put it on the master branch, so feel free to try it. The command is mesh2con, and I put a new text plane on the left, where the keypad keys / * - + can manipulate it. The code is here. Edit: included in v0.9.1-beta.5

[311-code]

Tried it out, but mesh2con still keeps the mesh far away and just attached to a red line, I was hoping thumbstick would somehow magically move it closer on z axis but does not. It's small because that is the scale it's supposed to be, but it's still stuck far away.

https://github.com/parkchamchi/DepthViewer/assets/23625562/640384af-1d1c-45ea-befe-a6cd5b96cecc

I was able to implement motion controls via new Meta all-in-on sdk and using the github samples here https://github.com/oculus-samples/Unity-StarterSamples. I am currently trying to convert to openxr. (stable diffusion generation of a marble bust + beit)

https://github.com/parkchamchi/DepthViewer/assets/23625562/80ca2b67-a34b-437f-b96c-53686c85d25f

Edit: Here is a better video of the new mesh2con setting after using set_camera_bg 0 255 0 0 for passthrough. Object far away behind wall.

https://github.com/parkchamchi/DepthViewer/assets/23625562/3cfc4599-66a5-48fa-a5e9-17c61d5fb177

Edit: Just read your release notes again The existing parameters, which are fit for the fixed-camera environment, would act weird. I am not too sure what you mean as I have it working in both VR at correct distance, and desktop mode is the same now in Unity build I just exported. I just had to modify the default distance and scale and those values in your mainbehavior and meshbehavior scripts. I only need to tweak it a little better in Unity and move the VR mesh down to where the desktop mode mesh is centered on the desktop GUI. (had to move meshver a bit in desktop mode)

[parkchamchi]

OK I put a new parameter MeshOffZ on the master branch. I didn't put the slider on the desktop GUI but put an entry on the World Plane (the one on the left). Just don't touch CamDistL and it'll work. (I suggest that it be fixed at ~~1~~ (0 seems better)). Edit: v0.9.1-beta.6

I am not too sure what you mean as I have it working in both VR at correct distance, and desktop mode is the same now in Unity build I just exported.

I was implying that when that mode is on, the existing parameter would not work as it may seem to. For example ProjRatio depends on the CamDistL, which was the distance between the camera and the mesh, but since the actual distance is independent of CamDistL it would not properly project the mesh.

[311-code]

Thanks, and that makes sense. I just got around to testing it and see the MeshoffZ parameter on the left but I can't seem to adjust it with any controls. I also also pressed every key on the keyboard.

Is there maybe a way to make so it adjusts this variable with the right thumbstick up and down on the motion controller? I think this would feel very natural, as the left thumbstick seems to rotate the mesh already. My intuition initially said I should use the right thumbstick to move the mesh closer the first time I tried mesh2con, since the left rotated.

In addition the A and B buttons for adjusting the scale up and down at a slow rate, and Y and X for 3D depth would basically make it perfect. (and perhaps in options menu a mesh2con (for VR) checkbox for new users would be helpful, maybe in options ability adjust set_camera_bg (for passthrough) option with R G B values you can type. The motion controls you implemented seem pretty nice so far and I actually prefer them over mine at the moment, because I have grab points and sometimes it's hard to find them. (though a way to remove red line when viewing the model while still using the motion controls would be ideal)

[parkchamchi]

The keypad (those near the num lock key) (/, *) selects the target parameter (denoted by > ...) and keypad (-, +) changes the value. Not the optimal control but it works. The keys are set in Keymapper.cs.

Accepting inputs from the control would be useful. The reason it accepts the left thumbstick already is cause it reads Input.GetAxis("Horizontal"), as here.

[311-code]

Ok I tried the keys you mentioned and it is working now. Though the red line seems to be a bit of an issue but this works very good in general. Is there a way to hide the menu? In passthrough mode I see the menu in the room behind the wall.

I tried to map the right controller to ParamLower and ParamIncrease like the Input.GetAxis("Horizontal") in the meshbehavior.cs, but it seems like the current setup does not explicitly differentiate between the left and right thumbsticks. I thought it was strange that in the input manager is set to use "all joysticks" for the horizontal and vertical input, but it does not rotate the mesh with the right stick in the build also like it says in the input manager, only the left. Which makes me wonder if it's actually using the input manager.

I tried checking all the Unity Input Manager settings and also the XR interaction toolkit code to see which was being used. So I'm not exactly sure how to do this.

[parkchamchi]

The red lines can be set in the Inspector menu of the Controller objects under XR Origin.

Seems like the other axes have to be set manually. On the Input Manager, increase the size by 2 and set like this: More info here

Now Input.GetAxis("HorizontalTurn") will response by the right thumbstick. Side note: my right thumbstick has slight input when it's in the neutral position, by 8%. Alignment problem?

Maybe I better move Keymapper.cs to the Input manager.

[311-code]

Side note: my right thumbstick has slight input when it's in the neutral position, by 8%. Alignment problem?

Yeah maybe defective controller or deadzone issue maybe the "dead" setting would help in input manager? Ok I see what you did on the Axis setting 4th joystick, would I do something like this in the meshbehavior.cs.. Not sure if this is right:

void Update()
{
    // Get vertical axis of right controller for scaling depthplane mesh
    float rightVInput = Input.GetAxis("VerticalTurn");

    // If thumbstick is moved downwards, increase the scale.
    // If thumbstick is moved upwards, decrease the scale.
    if (rightVInput < 0) ParamIncrease();
    if (rightVInput > 0) ParamLower();
}

[parkchamchi]

@brentjohnston Refer to this script.

[311-code]

I've been out of town but I had actually deleted that script since it was creating errors in Unity in this current project I am using meta all-in-one sdk for, the keyboard still seems to work though.

I made a new project and the error wasn't there. Now that the mesh2con works, I may just mess around with new project instead.

Before I do that, are there any other ways to do it quickly or it has to be done through that file? (The right thumbstick controlling the ParamIncrease and ParamLower / meshoffz)

[parkchamchi]

By quickly you mean changing the value? Well that script increases/decreases the value linearly, so if there be a acceleration value to it that may be done.

For right thumbstick do something like this:

if (Input.GetKeyDown(Keymapper.Inst.ParamPrev))
	_paramIdx--;
else if (Input.GetKeyDown(Keymapper.Inst.ParamNext))
	_paramIdx++;
else if (Input.GetKey(Keymapper.Inst.ParamLower))
	delta = -_paramInterval;
else if (Input.GetKey(Keymapper.Inst.ParamIncrease))
	delta = +_paramInterval;

delta += Input.GetAxis("HorizontalTurn") * Constant; //this

Or only for MeshOffZ:

const string targetParam = "MeshOffZ";

var zDelta = Input.GetAxis("HorizontalTurn");
if (zDelta != 0) {
	var origZ = _paramnames[targetParam];
	var newZ = origZ + zDelta * Constant;

	_dmesh.SetParam(targetParam, newZ);
}