SDL GPU: OpenXR integration

After taking a deep dive into the various graphics API init routines for OpenXR (and reading the entire core specification front to back), I think the only way to make this work cleanly and according to the OpenXR spec is if SDL fully wrapped the creation of your OpenXR Instance, System, and Session into the same function call that creates the SDL_GPUDevice*. There is just way too much intertwined with the core init code of the GPU device, since all the extensions tell you which GPU device/instance/adapter you must use for rendering with OpenXR.

Along with this, there should likely be an SDL_GPUCreateXrSwapchain function, which picks the best format to use, and returns the created swapchain, and also SDL_GPUEnumerateXrSwapchainImages, which returns the SDL_GPUTexture* for every swapchain image in the swapchain you need to render to. The SDL_GPUCreateXrSwapchain function can probably be omitted if desired, by having an SDL_GPUGetXrSwapchainFormats, which internally calls xrEnumerateSwapchainFormats and returns all the ones SDL_GPU can render to.

Init steps for each API

These are written from the perspective of SDL creating a GPU device from scratch (after it has determined which API it wants to use for VR)

XR_KHR_D3D11_enable

Create an XrInstance with the XR_KHR_D3D11_enable extension enabled.
Create the XrSystem
Call xrGetD3D11GraphicsRequirementsKHR with the created OpenXR instance and system to get the feature level and adapter to use when creating your D3D11 objects.
Create your D3D11 device and such
Create an XrSession with the next ptr of XrSessionCreateInfo being a XrGraphicsBindingD3D11KHR with the created ID3D11Device
Continue creating the SDL_GPUDevice* as normal

XR_KHR_D3D12_enable

Create an XrInstance with the XR_KHR_D3D12_enable extension enabled.
Create the XrSystem
Call xrGetD3D12GraphicsRequirementsKHR to get the minimum feature level and adapter to use.
Create your D3D12 device and a command queue
Call xrCreateSession with XrSessionCreateInfo.next pointing to a XrGraphicsBindingD3D12KHR with the created ID3D12Device and ID3D12CommandQueue
Continue creating the SDL_GPUDevice* as normal

XR_KHR_metal_enable

Create an XrInstance with the XR_KHR_metal_enable extension enabled.
Create the XrSystem
Call xrGetMetalGraphicsRequirementsKHR to get the MTLDevice to use
Create the GPU objects, including a command queue
Create an XrSession with the next ptr of XrSessionCreateInfo pointing to XrGraphicsBindingMetalKHR filled in with the created command queue.
Continue creating the SDL_GPUDevice* as normal

Note about Vulkan

From what I can see, you probably need to support both the XR_KHR_vulkan_enable and XR_KHR_vulkan_enable2 extensions, as some runtimes may only support one or the other.

XR_KHR_vulkan_enable

Create an XrInstance with the XR_KHR_vulkan_enable extension enabled
Create the XrSystem
Call xrGetVulkanGraphicsRequirementsKHR to get the minimum/maximum Vulkan version supported by the OpenXR runtime
Call xrGetVulkanInstanceExtensionsKHR to get the instance extensions to enable when creating your VkInstance
Create your VkInstance
Call xrGetVulkanDeviceExtensionsKHR to get the Vulkan device extensions to enable when creating your VkDevice
Call xrGetVulkanGraphicsDeviceKHR to get the VkPhysicalDevice to use
Create your VkDevice, pick a queue family+index
Create an XrSession with the XrSessionCreateInfo.next ptr pointing to a XrGraphicsBindingVulkanKHR with the VkInstance, VkPhysicalDevice, VkDevice, queue family, and queue index to use.
Continue creating the SDL_GPUDevice* as normal

XR_KHR_vulkan_enable2

Create an XrInstance with the XR_KHR_vulkan_enable2 extension enabled.
Create the OpenXR System
Call xrGetVulkanGraphicsRequirements2KHR to get the minimum/maximum Vulkan version supported by the OpenXR runtime
Call xrCreateVulkanInstanceKHR to create your VkInstance
Call xrGetVulkanGraphicsDevice2KHR to get the VkPhysicalDevice to use
Call xrCreateVulkanDeviceKHR to create the VkDevice to use
Select your queue family/queue index
Create the XrSession with the XrSessionCreateInfo.next ptr pointing to a XrGraphicsBindingVulkanKHR with the VkInstance, VkPhysicalDevice, VkDevice, queue family, and queue index
Continue creating the SDL_GPUDevice* as normal

Brief summary of the handles that need to be passed between OpenXR and SDL, mostly to give scale of just how intertwined the init of the GPU device and OpenXR session really are.

List of GPU handles needed to pass into OpenXR

ID3D11Device*
ID3D12Device*
ID3D12CommandQueue* (NOTE: I haven't used OpenXR with D3D12, but I from reading the spec, I believe this can be any D3D12 command queue)
MTLCommandQueue (it actually wants an "an Objective-C object that conforms to the MTLCommandQueue protocol", but I dont know enough metal/obj-c to know what that entails)

XR_KHR_vulkan_enable

XR_KHR_vulkan_enable2

PFN_vkGetInstanceProcAddr
const VkInstanceCreateInfo*
const VkAllocationCallbacks*
VkPhysicalDevice
const VkDeviceCreateInfo*
const VkAllocationCallbacks*

List of GPU handles returned by OpenXR, needed to be passed into the GPU API

ID3D11Texture2D*, returned by xrEnumerateSwapchainImages
ID3D12Resource*, returned by xrEnumerateSwapchainImages
MTLTexture, returned by xrEnumerateSwapchainImages. (once again it's actually a void*, and is defined as "texture is populated with a valid Metal texture to use, which must be able to be bridged casted to an Objective-C object that conforms to the MTLTexture protocol")
MTLDevice, returned by xrGetMetalGraphicsRequirementsKHR. The spec seems to force you to use a specific MTLDevice on metal, you can't create your own.
VkImage returned by xrEnumerateSwapchainImages
LUID for the D3D11/D3D12 adapter to use

XR_KHR_vulkan_enable

VkPhysicalDevice, like Metal, it seems that Vulkan with OpenXR also forces you to use a particular VkPhysicalDevice, and the application is unable to pick one on it's own.

XR_KHR_vulkan_enable2

The various OpenXR extensions also return some required info to init a functional GPU device for OpenXR usage

Minimum feature level for D3D11 device, returned by xrGetD3D11GraphicsRequirementsKHR.
Minimum feature level for D3D12 device, returned by xrGetD3D12GraphicsRequirementsKHR
(XR_KHR_vulkan_enable) Minimum/Maximum Vulkan version supported by the OpenXR runtime, returned by xrGetVulkanGraphicsRequirementsKHR
(XR_KHR_vulkan_enable2) Minimum/Maximum Vulkan version supported, returned by xrGetVulkanGraphicsRequirements2KHR

Sep 23 '24 04:09 Beyley

Agreed - in the nuclear case we can just add a property but it'd be nice to have a full list of what apps need to support VR with any given GPU API.

We could maybe kill two birds with one stone by having an OpenXR sample at https://github.com/TheSpydog/SDL_gpu_examples, and developing a gpu-xr PR to support it. My only experience with XR right now is input, for #4464, so I dunno how much help I'll be.

Sep 23 '24 11:09 flibitijibibo

it'd be nice to have a full list of what apps need to support VR with any given GPU API.

I've updated the issue with a full breakdown of the init process for every currently supported graphics API, turns out this will probably be a lot more involved than I originally thought. But should still be possible without exposing any raw GPU handles to the end user.

We could maybe kill two birds with one stone by having an OpenXR sample at TheSpydog/SDL_gpu_examples, and developing a gpu-xr PR to support it. My only experience with XR right now is input, for https://github.com/libsdl-org/SDL/issues/4464, so I dunno how much help I'll be.

I might fork off SDL and SDL_gpu_examples at some point, to try to actually get some code down for this, unless someone gets to it before me

Sep 24 '24 04:09 Beyley

I have started very early work on this here and here. Right now I have instance/system/session creation working on Vulkan with KHR_vulkan_enable2. (side note: my SDL is saying its forked off that other repo since i've forked SDL in the past for other stuff)

I have nothing visual to show yet, since I still gotta write the rest of the surrounding code in the example itself, and also write some swapchain related functions in SDL_gpu to let me render to the XR session swapchain.

Only after that can I actually start working on cleaning this up to upstream. (which includes stuff like making the OpenXR loader dynamically loaded)

I'm terrible with CMake, so forgive my dirty hacks and probable sins in the CMake patches to SDL :smiling_face_with_tear:

Sep 29 '24 05:09 Beyley

Is this still relevant today? I would be very interested in developing VR games next!

Nov 05 '24 11:11 corentin35000

Is this still relevant today? I would be very interested in developing VR games next!

I haven't worked on my fork in about a month or so, but I'm still interested in making this happen, I'm just very busy with other projects at the moment, feel free to contribute to my fork if you want to see this make progress. Right now it desperately needs a rebase

Nov 05 '24 21:11 Beyley

After some hacking today, i managed to get us all the way to rendering to the swapchain (with some hacks/hardcoded values)

big problem is that Monado's debug view is showing massive texture corruption (however the main simulated HMD's view is displaying correctly!) im assuming this is some UB im hitting with vulkan, but given the app crashes in renderdoc (and the fact I don't really know much vulkan), I'm not entirely sure yet how to debug it, but i've at least made some progress!

Dec 05 '24 10:12 Beyley

another update: i got it all working! top left is a "desktop view of the scene", rendered myself to an SDL window using SDL_gpu middle is Monado's debug interface, showing the composited image raw bottom right is the GUI used for controlling the QWERTY virtual HMD, showing a readback from the GPU of the composited image (should always match the debug interface's view)

not drawing anything pretty, but it's all working, aside from some vulkan validation issues on app close im not entirely sure how to solve yet.

Dec 06 '24 07:12 Beyley

Is this still the best available option?

Aug 20 '25 00:08 Starkium

Is this still the best available option?

Could you elaborate what your question means? OpenXR is the current most-supported standard for cross-platform and cross-vendor VR and AR experiences, with multiple compliant runtimes available on all platforms (except macOS, where there's only a single simulator runtime [Meta XR Simulator]).

Aug 20 '25 00:08 Beyley

Is this still the best available option?

Could you elaborate what your question means? OpenXR is the current most-supported standard for cross-platform and cross-vendor VR and AR experiences, with multiple compliant runtimes available on all platforms (except macOS, where there's only a single simulator runtime [Meta XR Simulator]).

Is this still the best way to work with openxr and SDL? was really hoping I didn't need to make numerous entry points for all different renderers and platforms. XR is the biggest offender. SDL supports OVR, or the other way around, but not openxr.

Looking around for the "best"/simplest option for tying together cross platform support / mostly uniform engine path for stuff.

Aug 20 '25 02:08 Starkium

Is this still the best way to work with openxr and SDL?

This issue just describes the internal to SDL work required to wire up OpenXR with SDL GPU.

SDL supports OVR, or the other way around, but not openxr

SDL has a video driver for OpenVR, but that's much different than what this issue is talking about, which is integration of OpenXR's graphical session initialization into SDL GPU to allow you to use SDL_gpu to render onto OpenXR swapchains.

Looking around for the "best"/simplest option for tying together cross platform support / mostly uniform engine path for stuff.

If you're dead set on SDL GPU, I'd recommend taking the fork of SDL in #11601, rebasing it, and using that. It supports Vulkan fairly well, Direct3D 12 with some probably easy to fix validation errors, and as of now, no Metal support. It's also tested to work on both Windows and Linux

Aug 20 '25 02:08 Beyley

I'll take a look at it, but I was considering using vulkan outside of sdl because of more features. Sorry, I didn't realize this was for sdl gpu api only.

Aug 20 '25 02:08 Starkium