Feature openxr

[Starcommander]

This pull-request adds the feature for using openxr in jme3. It is a working first implemention and some improvements are still necessary.

• Moving HMD (currently only rotating)
• Controllers
• ~~View perspective is not correctly~~

I tested it (and works) with OSVR-HMD on Debian 11 (Bullseye) with running monado-service. I also tested it (and works) with my HTC-VIVE.

To get it running:

• In main function modify app settings with: XrHmd.setRendererForSettings(appSettings);
• In simpleInitApp(): XrHmd.initHmd(this);
• Add your stuff into rootNode
• Start monado-service
• Start app.

[stephengold]

Given JME's staffing constraints, I worry about adding to the project's maintenance burden.

• Has adding OpenXR support been discussed at the JME Hub/Forum?
• Is there a good reason develop the jme3-xr library in the jmonkeyEngine repo, instead of as a separate project (like Blocks, Lemur, Minie, and ShaderBlow)?

[Starcommander]

@stephengold Here is a Forum, where openxr was discussed. https://hub.jmonkeyengine.org/t/solved-vr-launch-title-and-icon-in-steamvr/43706/14

When we have openvr included in jme3-vr (that needs steam binary blob "steamvr"), then why not openxr that is open source, and should be the successor of openvr?

[neph1]

I think it would be a great contribution to add openxr. Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

At the same time I share @stephengold 's concerns. The vr package hasn't had a proper maintainer since 2015 and has relied on sporadic contributions since.

I wonder if it would make sense to break it put of core to a separate project. It could still reside in the jme project, but not alongside the core packages. That way, updating it would be a little less sensitive.

Commenting on the pr itself, I think proper hmd translation and controller support would be minimum requirements for it to be considered. Otherwise the use case is very limited.

[jseinturier]

Integrating OpenXR to JMonkey is a good idea for me. However, i've some concerns regarding the proposed method:

• jme3-vr module has been thinked in order to provide a high level abstraction of existing VR API (at the beginig OpenVR and VRAPI). Integrating OpenXR should be achieved by extending the current jme3-vr module instead of removing it and providing an OpenXR exclusive module. At this time, OpenVR and VRAPI are supported and are binded by LWJGL. Removing these functionnalities from JMonkey can be problematic for end users.

• OpenXR provide Virtual Reality and Augmented Reality functionnalities. In the "real life" we are not offen use VR / AR at the same time so for me it is not relevant to create a new module that embed all the functionnalities. I think we could split VR / AR functionnalities within separated modules (jme3-vr, jme3-ar ?)

[richardTingle]

Very cool, I will take a look at this PR properly on Friday (and can test on an occulus quest). But I would vote yes to having openXR be an engine feature rather than a seperate library. It's a fundamental capability like android launch that might then have libraries build around it to produce advanced functionality.

I'm also pro this being seperate from the existing VR stuff because that was written in a very different time where there were many competing standards that needed to be abstracted over

[Ali-RS]

Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

From a quick look at this PR (in the gradle build), I do not see how the jme3-xr project relies on jme3-vr.

[neph1]

Adding just openxr as a separate project is probably not feasible as it relies on jme-vr.

From a quick look at this PR (in the gradle build), I do not see how the jme3-xr project relies on jme3-vr.

Damn. I thought I saw it was when I checked on my phone last night. Disregard that part.

[Starcommander]

@Ali-RS Yes, you are right, jme3-vr is not necessary to launch an app with jme3-xr. So, maybe jme3-xr can be a successor of jme3-vr in future. I will also still work on improvements of this project, for example controller support, and so on.

[richardTingle]

I've had a play with this on an Oculus Quest 2 on steam running with over virtual desktop and it all works (or the bits you said work work, being able to rotate my eyes but not move my eyes was a weird experience 😆).

First of all I'd like to say this is really impressive, when I tried to do this it was all so complicated I put it on the back burner.

Then I want to get some context for this. I don't want to make a bunch of nit-picky comments about naming and code structure if you're presenting this as the the major lift to get the core of XR working and then we can all collaboratively work on it. I know I'm all excited now; I did the action based openVR stuff so it might make sense for me to look at actions in openXR as they seem similar (I also have a vested interest in them being implemented similarly in JME as I'll want to upgrade Tamarin to be based on OpenXR and that will be easier if it's similar). I also though don't want to step on your toes

[richardTingle]

I had a go getting movement to work. I've kind of got it, there is a branch that builds on this here: https://github.com/richardTingle/jmonkeyengine/tree/openxr-withmove

It mostly works but there is something wrong with the field of view. It feels a bit like you're looking through a fish eye lens. Not sure why yet as I used the FOV reported by OpenXR

[stephengold]

When we have openvr included in jme3-vr (that needs steam binary blob "steamvr"), then why not openxr that is open source, and should be the successor of openvr?

From what I'm reading, JME will probably want to use OpenXR eventually. However, JME publishes feature releases about once per year, so developing it here means the new library probably won't be put into production until 2024. If you develop jme-xr as a separate project, you can release new versions as often as you want. Then when it's stable, we could easily incorporate it into JME and make it an official part of the Engine.

[richardTingle]

If we do want it in a seperate library I'd be happy to have it in Tamarin (as a Tamarin 2 that is only openXR and Tamarin 1 keeping legacy support for openVR) and then I'd maintain it

[richardTingle]

I've been working on actions and am making a fair bit of progress. Have got basic button presses registered and working (I mention this to avoid us duplicating work).

I have no idea why I'm getting the weird lensing effects when I turn my head and have given up trying to understand why (at least for now)

[stephengold]

Could you please convert this PR to a "draft"? That would make it obvious that the PR isn't ready to be integrated.

[Starcommander2]

Could you please convert this PR to a "draft"? That would make it obvious that the PR isn't ready to be integrated.

Yes, when I am back on Friday I will switch to draft. Unfortunately I have no access to my account currently.

[Starcommander]

@richardTingle You made some interesting progress regarding actions and some little changes in hmd-moving How would you suggest, we should continue the work? I would really like to see jme3-xr in next release of jmonkeyengine :wink:

[richardTingle]

@richardTingle You made some interesting progress regarding actions and some little changes in hmd-moving How would you suggest, we should continue the work? I would really like to see jme3-xr in next release of jmonkeyengine 😉

@starcommander yes, actions are going well, I have basic button presses and haptics working. Getting hand skeletons and hand poses working next; then I'll try to get tamarin's bound hands and grab interactions working with it. I much prefer openXR's approach to actions; suggested bindings specified in code, rather than openVRs manifest file that must be specified by an absolute path.

I found a method to get openXR to suggest eye positions and angles which is what I used in that hmd change.

On camera stuff I'm much less good, are you happy to tackle that? I couldn't get to the bottom of the lensing effects I was seeing. Do you see that too? That if you look directly at something it's in the right place but as you move your head it seems to move.

I'm fairly relaxed on it being a jme3-xr or incorporated into Tamarin (im less keen on a non Tamarin non jme library but thats just me being selfish as Tamarin is my library so if other people want that I'll go with the flow). 2024 does sound like a long time to wait though; that said an OpenVr to OpenXR migration looks like it would be quite easy for existing projects. I will at the very least want to keep the more opinionated bound hand, grab and lemur support within Tamarin.

[Starcommander]

I found a method to get openXR to suggest eye positions and angles which is what I used in that hmd change.

On camera stuff I'm much less good, are you happy to tackle that? I couldn't get to the bottom of the lensing effects I was seeing. Do you see that too? That if you look directly at something it's in the right place but as you move your head it seems to move.

When you found a method, then where do you use it? I only see changes, where you just removed the rotation+distance code. So, I also tested your modifications:

• No eye-rotation/eye-distance, so I can only see the same on left and right eye
• Moving HMD does still not work (does nothing)

Regarding the "lensing-effect" you mentioned:

• I think I can see it now, since the fieldOfView is now set to a value, that makes a better sense.
• But it has nothing to do with moving HMD, instead it appears when rotating the HMD, right?

[richardTingle]

@Starcommander That's odd. I definitely can move my head and the cameras move (and closing and opening alternating eyes gives different angles on scene, as you'd expect. This is all on the openxr-actions branch.

And in HelloOpenXRGL it sets the position and rotation for each eye

        viewPos.set(pos.x(), pos.y(), -pos.z());
        viewRot.set(orientation.x(), orientation.y(), -orientation.z(), orientation.w());
        
        Eye eye = viewIndex == 0 ? xrHmd.getLeftEye() : xrHmd.getRightEye();

        eye.setRotation(viewRot.inverse());
        eye.setPosition(viewPos);

I'm not very happy with those coordinate transformations, as JME and OpenXR are supposed to have the same coordinate system. I might try putting in the observer (the reference node from OpenVR implementation) and see if I can use that instead to get consistent directions with the axes.

But it has nothing to do with moving HMD, instead it appears when rotating the HMD, right?

True, its when you rotate your head that the lensing becomes apparent.

This is the test application I have been using (It only has suggested bindings for Oculus because that's what I've been using to test)

public class XRTestMain extends SimpleApplication{

    public static void main(String[] args){

        XRTestMain app = new XRTestMain();
        AppSettings settings = new AppSettings(true);
        XrHmd.setRendererForSettings(settings);
        app.setSettings(settings);
        app.start();
    }

    @Override
    public void simpleInitApp() {
        XrHmd.initHmd(this);

        for(int i = 0; i < 5; i++){
            Box b = new Box(0.1f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(i,0,0);
            rootNode.attachChild(geom);
        }
        for(int i = 1; i < 5; i++){
            Box b = new Box(0.05f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(0,i,0);
            rootNode.attachChild(geom);
        }
        for(int i = 1; i < 5; i++){
            Box b = new Box(0.1f, 0.1f, 0.1f);
            Geometry geom = new Geometry("Box", b);
            Material mat = new Material(assetManager,
                    "Common/MatDefs/Light/Lighting.j3md");
            geom.setMaterial(mat);
            geom.setLocalTranslation(0,0,i);
            rootNode.attachChild(geom);
        }

        rootNode.attachChild(checkerboardFloor(assetManager));

        DirectionalLight directionalLight = new DirectionalLight(new Vector3f(0.4f,-0.3f,0.6f).normalizeLocal());
        AmbientLight al = new AmbientLight();
        al.setColor(ColorRGBA.White.mult(0.3f));
        rootNode.addLight(al);
        rootNode.addLight(directionalLight);
    }

    boolean started = false;

    @Override
    public void simpleUpdate(float tpf) {
        OpenXRActionState state = getStateManager().getState(OpenXRActionState.ID, OpenXRActionState.class);

        if (!started) {
            Action teleport = Action.builder()
                    .actionName("teleport")
                    .translatedName("teleportTranslation")
                    .actionType(ActionType.BOOLEAN)
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().leftHand().xClick())
                    .build();


            Action trigger = Action.builder()
                    .actionName("fire")
                    .translatedName("fireTranslation")
                    .actionType(ActionType.BOOLEAN)
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().leftHand().triggerValue())
                    .withSuggestedBinding(OculusTouchController.PROFILE, OculusTouchController.pathBuilder().rightHand().triggerValue())
                    .build();

            Action haptic = Action.builder()
                    .actionName("haptic")
                    .translatedName("haptic")
                    .actionType(ActionType.HAPTIC)
                    .withSuggestAllKnownHapticBindings()
                    .build();

            Action pose = Action.builder()
                    .actionName("pose")
                    .translatedName("pose")
                    .actionType(ActionType.POSE)
                    .withSuggestAllKnownGripPoseBindings()
                    .build();

            ActionSet testActionSet = ActionSet.builder()
                    .name("test")
                    .translatedName("testTranslation")
                    .withAction(teleport)
                    .withAction(trigger)
                    .withAction(haptic)
                    .withAction(pose)
                    .build();

            ActionManifest manifest = ActionManifest.builder()
                    .withActionSet(testActionSet)
                    .build();


            state.registerActions(manifest, "test");


            started = true;
        }else{
            {
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "teleport", null);
                if(booleanActionState.hasChanged()){
                    System.out.println(booleanActionState.getState());
                }
            }
            {
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "fire", null);
                if(booleanActionState.hasChanged()){
                    System.out.println("fire"+ booleanActionState.getState());
                }
            }
            for(HandSide handSide : HandSide.values()){
                BooleanActionState booleanActionState = state.getBooleanActionState("test", "fire", handSide.restrictToInputString);
                if(booleanActionState.hasChanged()){
                    System.out.println("fire " + handSide + booleanActionState.getState());
                    state.triggerHapticAction("test", "haptic", 0.1f, 50, 0.2f, handSide.restrictToInputString);
                }
            }
        }
    }

    public static Geometry checkerboardFloor(AssetManager assetManager){
        Quad floorQuad = new Quad(10,10);
        Geometry floor = new Geometry("floor", floorQuad);

        Texture floorTexture = assetManager.loadTexture("Textures/checkerBoard.png");
        floorTexture.setMagFilter(Texture.MagFilter.Nearest);
        Material mat = new Material(assetManager,"Common/MatDefs/Misc/Unshaded.j3md");
        mat.setTexture("ColorMap", floorTexture);
        mat.getAdditionalRenderState().setFaceCullMode(RenderState.FaceCullMode.Off);

        floor.setMaterial(mat);
        Quaternion floorRotate = new Quaternion();
        floorRotate.fromAngleAxis(-FastMath.HALF_PI, Vector3f.UNIT_X);
        floor.setLocalRotation(floorRotate);
        floor.setLocalTranslation(-5,-1,5);

        return floor;
    }
}

There is definitely something wrong with the scene, the boxes seem like they are being looked at from the inside

[richardTingle]

I've got hand skeletons more or less working. Weirdly they are mirror images (So the left hand has a right hand and vice versa). Might be more coordinate incompatibilities between JME and OpenXR.

This shows the hand joints (each cube is a joint), the large green cube is the overall pose location

I don't want to go too much further without getting the cameras sorted out (especially given the coordinate transformations that seem to be being needed). Do you mind if we look at that in parallel and see what we both find?

[richardTingle]

I've asked a question about the lensing at https://hub.jmonkeyengine.org/t/openxr-vr-rendering-cameras-into-eyes/46837

[Starcommander]

@richardTingle I found out, why we see the boxes from inside. See at HelloOpenXRGL.java: glFrontFace(GL_CW); //Clockwise This should be: glFrontFace(GL_CCW); //Counter-Clockwise as in jme3 I will commit this modification now.

[richardTingle]

@Starcommander Nice! I bet that was why the hands were showing mirror imaged. I should be able to remove a bunch of nasty fudge factors from the actions stuff now which will put me in a much better place for getting tamarin hands working.

[Starcommander2]

@richardTingle: Hint: You have to disable GL_CULL_FACE in HelloOpenXRGL.java. Otherwise you will have a black screen, as offGeo is not rendered.

Am 18. Juni 2023 13:54:42 MESZ schrieb richardTingle @.***>:

@Starcommander Nice! I bet that was why the hands were showing mirror imaged. I should be able to remove a bunch of nasty fudge factors from the actions stuff now which will put me in a much better place for getting tamarin hands working.

-- Reply to this email directly or view it on GitHub: https://github.com/jMonkeyEngine/jmonkeyengine/pull/2012#issuecomment-1596116618 You are receiving this because you commented.

Message ID: @.***>

[stephengold]

What's the status of this PR? Is this still a work in progress, or has it stalled out?

[richardTingle]

I tried to figure out where the lensing effect was coming from, but in the end I gave up. I think to solve that I'd have to start from scratch to really understand what's going on. I'm not actively working on it, not sure if @Starcommander2 is

[Starcommander]

Is still work in progress. Last time i fixed the issue, that we see the boxes from inside.

Also I am very interested in the configuration (Hardware, OS, xr-runtime) that @richardTingle uses. Because I have moving not working currently, where it works for richardTingle. Seems there is a difference in xr-runtime, where I use monado-service on Debian11.

[richardTingle]

I'm on and Oculus Quest 2, over virtual desktop to a windows 10 PC using Steam VR as my xr-runtime

[richardTingle]

I have had another go at this. I started again from the openXR HelloOpenXRGL example making sure I understood what it was doing but taking what we learned from this attempt as well.

I think I've solved it this time. I found why I was getting lensing effects; the OpenXR runtime was requesting an asymmetrical field of view (with angleLeft != angleRight) but I was using the standard camera#setFrustumPerspective which doesn't support asymmetrical field of view but when I instead calculated a projection matrix and used camera#setProjectionMatrix it stopped lensing. It makes sense to me now that @Starcommander didn't see the lensing as presumably your headset didn't request an asymmetrical field of view (or al least not so heavily asymmetric) and so the problem didn't occur for you.

We were talking before about whether this should be in JMonkey proper or in a library. I did my attempt in a tamarin 2.x branch. Are we ok with that as an approach @Starcommander & @stephengold ?

My approach adopts the existing JMonkey OpenGL context rather than creating its own which means it can be a pure app state attached "whenever" in the applications running process rather than needing to be booted up before JMonkey.

An ultra minimal example application would look like this

public class MinimalOpenXr extends SimpleApplication {
    public static void main(String[] args) {
        AppSettings settings = new AppSettings(true);
        settings.put("Renderer", AppSettings.LWJGL_OPENGL45);
        MinimalOpenXr app = new MinimalOpenXr();
        app.setSettings(settings);
        app.start();
    }

    @Override
    public void simpleInitApp(){ 

        getStateManager().attach(new XrAppState());
        // below is optional to get actions working (including hand bones). A manifest needs to be created to describe the available actions
        getStateManager().attach(new OpenXrActionState(manifest(), ActionSets.TEST_SET));

        Geometry geom = new Geometry("Box", new Box(1, 1, 1)); 
        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setColor("Color", ColorRGBA.Blue); 
        geom.setMaterial(mat);
        geom.setLocalTranslation(new Vector3f(3,0,0));
        rootNode.attachChild(geom);  
    }
}

I already have these things working:

• Non distorted visuals
• Cameras tracking headset position
• Button press actions
• Hand Pose actions (aka hand positions)
• Hand Bone positions

I am now working on these

• Moving the observer (aka moving the offset between where the player is in the real world to where in the virtual world - teleporting etc)
• Getting the hand models to pair correctly to the bones. The OpenXR bones seem slightly different to the OpenXR ones so despite already having the bone positions this doesn't seem to immediately give me working hand models. I always thought the OpenVR bones were eccentric so if openXR has fixed this I think that's fine and I'll update my model's bones.

Once I get those working I'll start updating the TamarinTestBed examples to use this. Once that all works we'll be approaching parity with OpenVR

[stephengold]

We were talking before about whether this should be in JMonkey proper or in a library. I did my attempt in a tamarin 2.x branch. Are we ok with that as an approach @Starcommander & @stephengold ?

As far as I'm concerned, the decision whether to be an Engine library or a 3rd-party library is up to you. I see a lot of advantages to being a 3rd-party library, especially if the codebase is changing rapidly. But in principle I have no objection to making jme3-xr part of the Engine.