Godot 4 VR editor (EARLY WIP)

[BastiaanOlij]

Godot VR Editor

note This is still very early days and we're still weeks away from a minimally functional implementation but the basics are starting to work.

This PR introduces the VR editor capability for Godot. This functionality is implemented as a module that is only compiled into the editor build of Godot and can be excluded. This PR focusses on introducing the minimally functional implementation to run the VR editor on desktop. Below I've tried to provide a list of features in various catagories we're planning to implement in this PR including marking those currently done.

Future PRs will extent this capability.

While we're introducing editor settings to enable the VR functionality automatically, right now the VR editor is enabled through a command line switch. Simply run godot as follows:

godot -e --xr-mode on

For the first version of the editor, and the scope of this PR, the non-3D part of the editor will run in a virtual window the user can interact with. The 3D part of the editor will change into the player being present within the scene currently being edited and being able to perform the actions normally done in the 3D pane of the editor, within VR (i.e. actions such as selecting objects, moving them, rotating them, etc).

After this base is build we'll start a journey on adding pure VR alternatives to some of the editor features currently constraint within the 2D UI.

General setup

Most of the general setup to start Godot in VR mode is done.

• [x] Enable XR mode on startup of the editor and project manager
• [x] Initialise a VR version of the editor or project manager
• [ ] Enabled XR mode through project settings.

VR Avatar

The VR Avatar class handles the user presence within the 3D scene.

It features a hud function that allows us to anchor UI elements in place. To combat motion sickness this UI follows the player around instead of being snapped in place. Rotation of these UI elements around the user will be controllable by the user themselves.

The users hands are currently tracked as well with both a "poke" feature (touching UI elements with ones index finger) and raycast interaction, both mimicking mouse functionality on "VR Windows" they interact with. Only left or right hand raycast is active at a given time, this is switched automatically when the user pulls the trigger on the desired hand.

"Grab" logic will also be added that allows the user to grab certain elements, this will be one of the options the user will have to re-arrange objects within the scene being edited.

A virtual keyboard is also embedded for the avatar (see VR keyboard)

• [x] Setup basic avatar with hands
• [x] Implement a hud system to anchor 2D UI elements to
• [ ] Implement system to reposition hud system (as an alternative to auto rotation which didn't work nice, auto positioning will be kept)
• [x] Implement poke logic so player can interact with UIs by touch
• [x] Implement raycast logic so player can interact with distant elements
• [ ] Implement grab functionality
• [ ] Implement something nicer to visualise hands
• [ ] Implement method for moving around
• [ ] Implement scene interaction logic

VR Window

The VR window class allows us to show 2D UIs as virtual windows in our headset. At some point we need to look into integrating this with the normal Window class in Godot so we can open up popups properly.

By default our interaction with raycast is setup as:

• Trigger acts as the left mouse button
• A button acts as right mouse button

Touch only triggers left mouse button.

• [x] Implement basic window support in VR
• [x] Implement mouse emulation based on poke/raycast from avatar
• [x] Add alternative select functionality (right click through A button)
• [ ] Add logic to allow throttling of the 2D interface to limit overhead of rendering it.
• [ ] Add movable/resizable borders

VR Keyboard

As in VR the user is likely standing or even when seated unlikely has visibility of their keyboard, being able to enter text requires a virtual keyboard.

The current implementation is a rudimentary keyboard implementation that emits Godot key events like a physical keyboard. I may redo the keyboard to a UI that is more suited to a VR environment but for now it suffices

• [x] Implement basic keyboard
• [ ] Allow modifiers of keyboard (ALT/CTRL)
• [ ] Fix bug that key events aren't working properly
• [ ] Internationalize keyboard

VR Project manager

The project manager has been exposed in its entirety through a single VRWindow. It is a bit of a gimmick on desktop to have this in VR but for the stand alone version this will be important to have working. It is also an easy test bed for a lot of the functionality we are building. The screenshot does not do it justice, it's much sharper inside of the headset. For now the environment is just a square for the floor, this is better then having the user float in space.

• [x] Implement a VR version of the project manager
• [ ] Make project manager work now that it defaults to OpenGL3, either by not doing so if VR is enabled or by giving @dsnopek a hand to get VR support on OpenGL3 over the line :)
• [ ] Make the UI work with the keyboard
• [ ] Create a nicer environment to keep the user grounded
• [ ] Alternatively, turn on pass through mode if supported and disable environment.

VR Editor

This is where the fun starts, it's still early days at the moment. The design of the VR editor is relatively straight forward. The user is embedded within the current scene we're editing, parts of the desktop UI are reused in floating windows to give access to scene data and commonly needed actions such as navigating the scene tree, adding nodes, etc.. A version of the gizmo system will be implemented for VR that allows the user to manipulate objects in the scene.

• [x] Implement basic setup for the VR editor
• [ ] Add the 2D editor UI as a window (WIP)
• [ ] Implement using raycast on controllers to select elements in the scene
• [ ] Implement gizmo system to interact with selected object
• [ ] To Be Determined.

Known issues:

• Currently keyboard is not functional, it emits key events but they are not being received by the field with focus
• If a VR game is being edited, the XRCamera3D node of the game is being updated with tracking data.
• The VR Avatar setup is currently rendering in the desktop 3D viewer, fun to watch but not good.
• Exiting out of Godot doesn't properly cleanup and can crash.

[BastiaanOlij]

Ok, capturing this in a comment as well as the OP description will change in time to reflect just the end result.

Starting to get the basics of the UI up and running in VR we found out pretty quickly that re-using GUI components of the editor UI was a no-go. A lot of parts of the editor were designed under the assumption there would only be one copy. Things like the scene dock (showing our scene tree), property inspector (showing the properties of the selected node), etc. are all singletons to make it easy to react to the users actions.

Reproducing the same UIs in the VR editor while it runs side by side with the desktop editor means we need two instances of these UI elements which breaks the singleton model. This means we either:

• need to do extensive rework on the existing editor to make multiple instances of these elements work
• need to build copies of these components from scratch introducing an immense amount of work and increase the workload of maintaining multiple copies of the same logic.

After discussing with @m4gr3d on the XR chat we came to the conclusion this was a foolish direction to take, especially seeing we already want to run the VR editor stand alone on mobile hardware like the Quest. Building the VR editor purely as a stand alone feature, even on desktop, greatly simplifies the task at hand and will allow us to reuse many components of the VR editor. There are a few challenges as we'll need to have our current editor and the VR editor share a common base class that virtualises the editor and then change a number of places that access the editors singleton, to go through the virtual base class, but that can be overcome.

CC: @reduz @akien-mga

[BastiaanOlij]

A note on this branch currently being based on an earlier commit of master, a recent change has made the project manager default to OpenGL3, currently we don't support VR in OpenGL3 yet, this is coming shortly. We will either for the time being force Vulkan for the VR project manager or I won't rebase this until VR support is ready.

CC: @dsnopek @clayjohn

[HeadClot]

Hey - got a few questions about the Standalone VR Editor.

1. Will this work with the recently added support for doubles?
2. Will we be able to use this VR Editor on standalone android powered VR devices like the Meta Quest 2, Pico 4, etc.?
3. Will we be able to pose and animate characters, props, etc in this VR Editor Mode?

Either way super excited and I cannot wait to test this out :)

[dsnopek]

currently we don't support VR in OpenGL3 yet, this is coming shortly

@BastiaanOlij Here's the start of my PR adding OpenGL support to OpenXR: #67775

[BastiaanOlij]

Hey - got a few questions about the Standalone VR Editor.

1. Will this work with the recently added support for `doubles`?

Can't see why not, though with the footnote seeing question number 2, as I understand it mobile GPUs may be less happy with doubles but I'm not sure if that applies to this particular issue.

2. Will we be able to use this VR Editor on standalone android powered VR devices like the Meta Quest 2, Pico 4, etc.?

That is the long term plan but at the moment the performance isn't there yet.

3. Will we be able to pose and animate characters, props, etc in this VR Editor Mode?

Probably not in the near future but in the long run we hope to get the entire (3D) editor experience to work in VR. Time will tell if we can make it.

[BastiaanOlij]

Small update for the latest changes I pushed. I've had to focus on a few other bits last week so most progress was made this week. After talking to @reduz we decided to change the approach somewhat so I've been doing a lot of refactoring this week.

Things changed so far are:

• VR editor now fully works stand alone, when --xr-mode on is added to the startup switch the main viewport becomes the XR viewport and the 2D interfaces are all placed in XR windows
• XR windows can now be curved as this gives a more pleasant viewing experience to the user
• Completely reworked the pointer logic into something more flexible (keyboard input is still broken, next on the list to fix)
• The normal editor interface is now contained in a single viewport/XRWindow. We'll be focusing on interacting in the 3D environment first. As we implement specific VR features of the editor we'll end up disabling/hiding stuff in the desktop one.
• Auto centering has been removed as this resulted in the editor always blocking our view. Working on a different UI for this.
• project manager in VR is currently forcing Vulkan, with @dsnopek work merged will be trying to make it run in OpenGL soon.
• Need to further clean up some of the windowing code that is no longer needed

[BastiaanOlij]

@dsnopek I tried turning off the workaround I added to force Vulkan if we're running the project manager. I can see it render upside down on the screen but don't get anything in the headset. Is there a step I'm forgetting on OpenGL?

[BastiaanOlij]

Had a few other priorities drop on my lap so haven't made a whole lot of progress, but fixed various bugs and added right click support (A button).

Worked on trying to solve the keyboard issue but while I made a bunch of improvements so the keyboard itself now works, still in the dark why the keyboard events don't end up where they should go (i.e. trying to rename a project, the text edit field ignores keyboard input, but it also ignores keyboard input from a real keyboard).

[intrepidOlivia]

Hey there, I'm looking for some troubleshooting steps. As soon as I start the VR editor, the scene loads in VR and I can see the editor and keyboard, but I can't use my hands, and my console is spammed with this message:

OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]

This isn't a super descriptive error message, do you know what I can try to adjust to make this work?

[m4gr3d]

Hey there, I'm looking for some troubleshooting steps. As soon as I start the VR editor, the scene loads in VR and I can see the editor and keyboard, but I can't use my hands, and my console is spammed with this message:

OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]
OpenXR: failed to sync active action sets! [ XR_ERROR_RUNTIME_FAILURE ]

This isn't a super descriptive error message, do you know what I can try to adjust to make this work?

@intrepidOlivia This issue should have been resolved by the latest push.

[BastiaanOlij]

@m4gr3d just fixed a compile error. note that https://github.com/godotengine/godot/pull/71830 got merged so some of the access methods got changed and may impact further work on your side.

[BastiaanOlij]

Merged in latest changes from @m4gr3d and squashed everything into a single commit.