vkd3d-proton icon indicating copy to clipboard operation
vkd3d-proton copied to clipboard

Resizable BAR / SAM tanking CPU performance on VKD3D

Open Faugus opened this issue 2 years ago • 20 comments

RX 6800 XT / 5600X

I've been doing some tests and I got the conclusion that Resizable BAR / SAM hurts the CPU perfomance when running games on VKD3D.

This problem doesn't happen on DXVK or Vulkan Native. Also this issue doesn't happen on Windows.

VKD3D games I tested: World of Warcraft, Uncharted 4, Spider-Man, Spider-Man Miles Morales, Resident Evil 2 Remake, Resident Evil Village...

I posted on Mesa's GitLab some tests and comparissons: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7993#note_1702565

Faugus avatar Jan 30 '23 17:01 Faugus

I can confirm the findings in Uncharted 4 on an Ryzen 3700x + RX7900XTX.

While SAM is enabled the game is weirdly CPU bound, weirdly because all cores show 40% usage with no core exceeding that. Disabling SAM (in the BIOS) leads to a much more healthy per core usage of 80% and a framerate uplift of +56% on my system.

Tested on an ASUS B450F, 32GB RAM, vkd3d 2.8. No OC or similar.

Mershl avatar Feb 20 '23 22:02 Mershl

Copied my results (Ryzen 3700x + RX7900XTX, vkd3d-proton 2.8) from mesa Gitlab:

3440x1440 Ultra Uncharted 4
SAM ON + VKD3D_CONFIG=no_upload_hvv 78 FPS
SAM ON 50 FPS
SAM OFF 78 FPS

I used the heaviest location I could fine for the test (most of the time FPS are higher, but gap stays about the same)

Turning on/off Above 4G decoding and Resizable BAR individually did not show a difference so I merged them into "SAM ON". Reported BAR is Detected VRAM RAM=24560M, BAR=32768M for SAM ON, Detected VRAM RAM=24560M, BAR=256M for SAM OFF.

no_upload_hvv could become a default config for Uncharted 4 until the issue is resolved. Though we have no Nvidia results for now.

Mershl avatar Feb 21 '23 14:02 Mershl

Also affects AoE 4 https://github.com/HansKristian-Work/vkd3d-proton/issues/1323, Death Stranding https://github.com/HansKristian-Work/vkd3d-proton/issues/864, and FFVII Remake (only during certain attacks during the Airbuster Boss fight it drops to 20 fps without no_upload_hvv)

mbriar avatar Feb 21 '23 14:02 mbriar

Just tried VKD3D_CONFIG=no_upload_hvv with Uncharted and it fixed the issue!

I got a small FPS hit, but now SAM is working properly.

Uncharted Menu Ingame
SAM ON + VKD3D_CONFIG=no_upload_hvv 141 FPS 138 FPS
SAM ON 146 FPS 88 FPS
SAM OFF 132 FPS 120 FPS

Faugus avatar Feb 21 '23 15:02 Faugus

but now SAM is working properly.

From my understanding using no_upload_hvv you're keeping vkd3d from using any BAR memory. So it's not actually "working", but you're avoiding this title-specific performance hit.

I'm really surprised you're seeing a benefit comparing ON + no_upload_hvv vs. OFF. I was not able to reproduce that.

Mershl avatar Feb 21 '23 15:02 Mershl

Do you have Resizable BAR option ON in BIOS?

If you do, try to disable it.

Leave only Above 4G decoding ON.

Faugus avatar Feb 21 '23 15:02 Faugus

Did a quick re-test. Tested on an ASUS B450F with this months latest BIOS (v5003).

3440x1440 Ultra Uncharted 4
Above 4G decoding + Resizeable BAR + VKD3D_CONFIG=no_upload_hvv 78 FPS
Above 4G decoding + Resizeable BAR 50 FPS
Above 4G decoding + VKD3D_CONFIG=no_upload_hvv 78 FPS
Above 4G decoding 51 FPS
everything OFF 78 FPS

Mershl avatar Feb 21 '23 16:02 Mershl

Maybe you're not seeing difference because you're running at 4K.

I'm running at 1440p.

Faugus avatar Feb 21 '23 16:02 Faugus

Here's a comparison running the game with ReBAR OFF, ReBAR ON and ReBAR ON + VKD3D_CONFIG=no_upload_hvv https://www.youtube.com/watch?v=78XakTs-3bI

VKD3D_CONFIG=no_upload_hvv - Blocks any attempt to use host-visible VRAM (large/resizable BAR) for the UPLOAD heap. May free up vital VRAM in certain critical situations, at cost of lower GPU performance. A fraction of VRAM is reserved for resizable BAR allocations either way, so it should not be a real issue even on lower VRAM cards. https://github.com/HansKristian-Work/vkd3d-proton

Faugus avatar Feb 25 '23 22:02 Faugus

I too am seeing this in Uncharted 4, fps in main menu explode from 90fps CPU bound to 160fps GPU bound. Doesn't happen on Windows with native D3D12, or at least it's fast enough to be always GPU bound there.

aufkrawall avatar Mar 18 '24 17:03 aufkrawall

Dragon's Dogma 2 also suffer from this.

with AMD CPU 3700x on main menu ~40fps without VKD3D_CONFIG=no_upload_hvv
~60fps with VKD3D_CONFIG=no_upload_hvv

Billli11 avatar Mar 22 '24 10:03 Billli11

Nvidia RTX 407 0- Control suffer from this too. Having ReBar enabled limits the GPU usage to ~65% in Control, depending on location. Disabling ReBar makes the GPU use all power (99%) and adds +15FPS in Control.

What's weird is that VKD3D_CONFIG=no_upload_hvv doesn't seem to do anything for this game, I have to disable Rebar in Bios to actually see a difference. Having rebar off in BIOS is not an option since it seem to hurt performance in other games.

shelterx avatar Mar 26 '24 14:03 shelterx

In Horizon Forbidden West it's the opposite.

3440x1440, Very High Preset, 5800x3D+7900XTX VKD3D_CONFIG not set: 105FPS VKD3D_CONFIG=no_upload_hvv: 94FPS

If no general solution can be found it might make sense to create game-specific no_upload_hvv defaults in device.c where it helps performance in most cases.

Mershl avatar Mar 31 '24 14:03 Mershl

Yes, I agree. However there might be different results depending on vendor and/or driver. The odd bird here is Control. I hope they figure that one out.

shelterx avatar Mar 31 '24 14:03 shelterx

One thing I noticed when having VKD3D_CONFIG=no_upload_hvv set is that a gameplay recording becomes jittery/stuttery.

Faugus avatar Mar 31 '24 14:03 Faugus

I can confirm this bug, without VKD3D_CONFIG=no_upload_hvv DD2 and some other games are unplayable.

Operating System: CachyOS Linux KDE Plasma Version: 6.1.3 KDE Frameworks Version: 6.4.0 Qt Version: 6.7.2 Kernel Version: 6.10.1-3-cachyos (64-bit) Graphics Platform: X11 Processors: 32 × Intel® Core™ i9-14900HX Memory: 31.1 Gio of RAM Graphics Processor: NVIDIA GeForce RTX 4090 Laptop GPU/PCIe/SSE2 Manufacturer: Alienware Product Name: Alienware m18 R2

Cardiacman13 avatar Jul 25 '24 22:07 Cardiacman13