[DRAFT] Add automated tests using Meta XR Simulator

[dsnopek]

trafficstars

The goal of the PR is to add the structure for automated tests running on GitHub Actions using Meta XR Simulator.

With the XR Simulator, it's possible to record inputs and periodically take screenshots, which both (the inputs and the images) get saved to a .vrs file. Then we can have the XR Simulator play back those inputs, taking screenshots at the same moments, recording into a new .vrs file. Then using pyvrs and a Python script provided with the XR Simulator, we can compare the screenshots, generate diffs, and decide whether they match or not based on a threshold of similarity.

Ultimately, I'd like to have a sample project and a .vrs file for testing each feature that can be tested via the XR Simulator (which includes passthrough along with the Scene API and Spatial Anchors, using synthetic environments, which is pretty cool).

However, the goal of this PR is just to figure out the structure, including one example .vrs file.

This is marked as a draft, because there's still a few things to figure out:

• Running the tests with a working Vulkan driver. I tried using SwiftShader as a CPU-based Vulkan implementation (the code is still commented out in the PR), but while this worked for Godot, the XR Simulator refuses to start an OpenXR session because there is not a "suitable" graphics device (I forget the exact error message). So, right now I'm running this with a self-hosted GitHub runner on one of my Windows machines. If we're going to stick with a self-hosted runner with a real GPU, I'd like to get it running in the cloud somewhere first.
• I'm using the build_profile PR. This makes build times much faster, but I'm not sure it's ready to be merged - see https://github.com/GodotVR/godot_openxr_vendors/pull/149 - if this PR ends up being ready first, I'll rebase that PR out of this one.
• I'm using a custom Windows build of Godot master. Once Godot 4.3-beta1 is out, we should be able to use that, but using Godot 4.3-dev6 crashes at the moment.

[BastiaanOlij]

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

[dsnopek]

Neat, I think the only thing we need to be careful off is that we don't put ourselves in a jam when upstream changes in Godot cause false positives in testing and we find ourselves unable to merge features because someone changed something in the rendering engine and missed the impact on XR (happens sadly all to often).

Since we can play them back and automatically record the new outcome, I think we could fairly easily make a script that just re-records all the .vrs files when there are rendering changes. We should probably pin a specific Godot version, and then when we update that, we can run that script if necessary.

[dsnopek]

I spent a little more time trying to make SwiftShader work, but still haven't managed it. So I don't lose track of it, this is the error that appears in the console when Godot tries to start an OpenXR session when using SwiftShader:

[XrSim][00000.146735][E][arvr\projects\openxr_simulator\src\sim_xrapilayer_rendering_vulkan.cpp:419] Failed to find a suitable GPU

We then get these messages from Godot on the following lines:

OpenXR: Failed to obtain vulkan physical device [ XR_ERROR_RUNTIME_FAILURE ]
ERROR: Condition "!device_retrieved" is true. Returning: ERR_CANT_CREATE
   at: _initialize_devices (drivers/vulkan/rendering_context_driver_vulkan.cpp:475)
ERROR: Condition "err != OK" is true. Returning: err
   at: initialize (drivers/vulkan/rendering_context_driver_vulkan.cpp:591)
ERROR: Unable to create DisplayServer, all display drivers failed.

However, Godot itself (even the editor) seems to run fine with SwiftShader, albeit very slowly. :-) It's the XR Simulator that seems to have a problem with SwiftShader.

[dsnopek]

I've updated the PR to point at Godot 4.2-beta1, and removed the dependency on PR https://github.com/GodotVR/godot_openxr_vendors/pull/149.

So, as soon as this passes tests, it'll finally be ready for review and merging!

[dsnopek]

This one is passing CI and ready for review now!

[fire]

We discussed how the vulkan code in xr simulator didn't like swiftshader and how we could check it and maybe create a minimal reproduction project (pull request) for someone to look at.