Lag/increased latency in VR applications

[Firnor]

I am currently using Falcor to develop a VR application utilizing the new DXR features. Although everything works more or less out of the box, I noticed increased latency (seems like 1 frame lag) when rendering to an HMD. This occurs in my application as well as in the StereoRendering sample. When shaking the head the scene seems to be a bit behind. This is especially easy to observe when looking at the chaperone while shaking since the scene moves relative to the static chaperone.

My initial guess was, that rendering is delayed by 1 frame. In fact, the command queue of the render context executes the command list from onFrameRender after I submit the VrFbo in onFrameRender. However, I moved the VrFbo submit call after the present call in frameRender (Sample) and it changed nothing. According to the OpenVR wiki, there's no need for synchronization between the execture command lists call and the VR framebuffer submission (see OpenVR wiki, lines 822 - 836)

I already checked all my SteamVR settings, I don't have this issue in any other VR app or game.

FYI: I use an HTC Vive (not pro), RTX 2080Ti with the newest graphics driver (425.31), SteamVR (1.3.22) and Falcor (3.2.1).

[nbentyNV]

There shouldn't be any frame lag. How do you the displayed image is 1 frame late?

If it's just feels delayed, it may be due to CPU latency. I suspect what happens is:

1. At the beginning of onFrameRender() you update the HmdCameraController (possibly indirectly by calling Scene::update()
2. You record a large command list. Note that this doesn't submit anything to the GPU, only records commands
3. present() is be called. This is where the command queue is being submitted

If step 2 takes a long time (CPU time), then by the time you get to present() the HMD data will be outdated. The correct solution is to reverse steps 1 and 2 - first create the command list and right before submitting the queue update the HMD data.

Unfortunately, there's no easy way to do it with Falcor. Recording commands also captures the state of the buffers - including their GPU address - and updating a buffer's data re-allocates the buffer first.

You can add a flag to ConstantBuffer::uploadToGpu() which will perform an update in-place (no reallocation). If you do that, what you'll need to do is:

1. Record all the commands
2. Update the HmdCameraController
3. Update the ConstantBuffer data and explicitly call ConstantBuffer::uploadToGpu()

For step 3, you won't be able to use the internal constant buffers, since those are versioned and cached. Just create your own buffer.

[Firnor]

It's just a guess that there is 1 frame latency. For testing, I delayed the rendering by 1 frame and it added a similar amount of lag. Like I mentioned before, when looking at the chaperone and shaking the head, the complete scene is a bit behind and moves relative to the chaperone.

You are correct, that's the order of steps I perform. The only weird thing is that this happens with small and large scenes (short and long CPU time respectively) but I assume that due to VSync it has this static amount of CPU time (11.1ms for 90Hz) and therefore it introduces the lag. Actually, I thought that the reprojection in OpenVR would do the trick, but seems like this is not enough.

I will definetly try what you recommend and then share my insights. Luckily, I only have to put tracking data into my own buffers, so that should not be too complicated to do.

[Will-VW]

I am encountering the same issue and it's made the Falcor VR mode entirely unusable to anyone remotely sensitive to motion sickness.

Even the stock example stereo render scene with your suggestions applied doesn't fix the issue.

I'm running at 144Hz locked at about 30% GPU usage on a 2080 Ti and no more than 25% CPU usage on a 9900k.

[Firnor]

I also tried the suggested solutions which unfortunately didn't work. Next guess was that triple buffering might cause the issues due to the frames in flight, but it didn't change anything when setting it to double or single buffering. When compiling the OpenVR example from their Github it works perfectly fine. I noticed, that the order of the important API calls is not the same (WaitGetPoses, ExecuteCommandList, etc.) but changing this also did not do the job. Updating to the newest OpenVR version also didn't work. For now, I learned to live with the lag but the biggest issue is when the app drops below 90 FPS (Vsync on HTC Vive) it should auto-enable motion smoothing and be at least somewhat ok, but it is extremely laggy and not very useful for me right now (working on my Master's Thesis).

[nikowiss]

Same for me first but at least I could avoid this sliding/swimming behaviour mentioned by @Firnor which gives you instantly motion sickness.

I don't have a full solution because I haven't identified the actual problem but I will list the points I've changed:

• I got rid of the VrFbo and used a simple Fbo with two color targets
• I don't use the submitToHmd(pContext) method. I use the direct method submit(...) of the VRSystem class instead and submit color target 0 and 1 after both are rendered.
• I'm using a deferred rendering
• I'm using two render passes (the bad naive way - I know it's not perfect but it's enough for my current context)

With this setup I'm able to run small scenes totally smooth using an Oculus Rift on a RTX 2070. The Arcade scene also works almost fine, although a few frame drops appear sometimes - but real frame drops. Not nice but still better than the swimming effect.

[Firnor]

Thank you very much @nikowiss, I will give it a try as soon as I can. Would be great if we could find out what causes these issues.

[Firnor]

I managed to find out what is causing this swimming behavior: As I noticed in my initial post, the example code in the OpenVR repository does use a different order of important API calls. Right now the order in Falcor is (in the StereoRendering sample):

1. WaitGetPoses
2. Record command lists
3. Submit VR Fbo
4. ExecuteCommandLists and GPU signal
5. Present

That way, it seems like OpenVR is submitting the old textures (1 frame behind) since the new rendering commands are not executed yet.

The order as given by the OpenVR example code is:

1. Record command lists
2. ExecuteCommandLists and GPU signal
3. Submit VR Fbo
4. Present
5. WaitGetPoses

My solution was to introduce 2 new callbacks to the renderer class: OnPrePresent and OnPostPresent. OnPrePresent is called right after ExecuteCommandLists and before Present (Device::present before apiPresent) and OnPostPresent is called at the end of Device::present. Now I can submit my VR textures in OnPrePresent and WaitGetPoses (VRSystem::refresh) in OnPostProcess to achieve the same order as in the OpenVR example and get rid of the lag. Additionally to that, it is currently not possible to use the VvFbo class for submitting the textures since they introduce an additional copy which happens after ExecuteCommandLists and therefore is ignored. To fix this I moved the copies to an explicit call (prepareSubmit) which is invoked at the end of my onFrameRender callback.

@nbenty I created a pull request to fix these issues (#211).