Requirements for DXVK support..

[oscarbg]

Hi, this is just a report of "things" missing in MoltenVK to support DXVK now that Wine supports Vulkan on MacOS via MoltenVK (I use v1.0.12 included in Vulkan MacOS SDK 1.0.77).. I tested DXVK 0.6.3 on Wine 3.13 with Vulkan support enabled on MacOS and simple programs like included d3d11-compute or d3d11-triangle failed due to unsupported extensions so I checked logs and seems it needs this extensions (as of 0.6.3):

From: https://github.com/doitsujin/dxvk/blob/master/src/dxvk/dxvk_extensions.h we see requirements for DxvkInstanceExtensions: VK_KHR_GET_PHYSICAL_DEVICE_PROPERTIES_2 the other two are emulated by Wine already..

for DxvkDeviceExtensions it needs: VK_KHR_DEDICATED_ALLOCATION_EXTENSION_NAME, VK_KHR_DESCRIPTOR_UPDATE_TEMPLATE_EXTENSION_NAME, VK_KHR_GET_MEMORY_REQUIREMENTS_2_EXTENSION_NAME,
VK_KHR_IMAGE_FORMAT_LIST_EXTENSION_NAME,
VK_KHR_MAINTENANCE1_EXTENSION_NAME VK_KHR_MAINTENANCE2_EXTENSION_NAME VK_KHR_SHADER_DRAW_PARAMETERS

I modified code of this exts from DxvkExtMode::Required to DxvkExtMode::Optional but then it complains it can't create a D3D device with support for any feature levels ranging from D3D_FEATURE_LEVEL_9_1 to D3D_FEATURE_LEVEL_11_1..

it's because needs features missing in MoltenVK (as of 1.0.12 in Vulkan SDK 1.0.77): code checking feature level is here: https://github.com/doitsujin/dxvk/blob/master/src/d3d11/d3d11_device.cpp `

if (featureLevel >= D3D_FEATURE_LEVEL_9_1) { enabled.depthClamp = VK_TRUE; enabled.depthBiasClamp = VK_TRUE; enabled.fillModeNonSolid = VK_TRUE; enabled.pipelineStatisticsQuery = supported.pipelineStatisticsQuery; enabled.sampleRateShading = VK_TRUE; enabled.samplerAnisotropy = VK_TRUE; enabled.shaderClipDistance = VK_TRUE; enabled.shaderCullDistance = VK_TRUE; enabled.robustBufferAccess = VK_TRUE; }

if (featureLevel >= D3D_FEATURE_LEVEL_9_2) {
  enabled.occlusionQueryPrecise                 = VK_TRUE;
}

if (featureLevel >= D3D_FEATURE_LEVEL_9_3) {
  enabled.multiViewport                         = VK_TRUE;
  enabled.independentBlend                      = VK_TRUE;
}

if (featureLevel >= D3D_FEATURE_LEVEL_10_0) {
  enabled.fullDrawIndexUint32                   = VK_TRUE;
  enabled.fragmentStoresAndAtomics              = VK_TRUE;
  enabled.geometryShader                        = VK_TRUE;
  enabled.logicOp                               = supported.logicOp;
  enabled.shaderImageGatherExtended             = VK_TRUE;
  enabled.textureCompressionBC                  = VK_TRUE;
}

if (featureLevel >= D3D_FEATURE_LEVEL_10_1) {
  enabled.dualSrcBlend                          = VK_TRUE;
  enabled.imageCubeArray                        = VK_TRUE;
}

if (featureLevel >= D3D_FEATURE_LEVEL_11_0) {
  enabled.shaderFloat64                         = supported.shaderFloat64;
  enabled.shaderInt64                           = supported.shaderInt64;
  enabled.tessellationShader                    = VK_TRUE;
  // TODO enable unconditionally once RADV gains support
  enabled.shaderStorageImageMultisample         = supported.shaderStorageImageMultisample;
  enabled.shaderStorageImageReadWithoutFormat   = supported.shaderStorageImageReadWithoutFormat;
  enabled.shaderStorageImageWriteWithoutFormat  = VK_TRUE;
}

if (featureLevel >= D3D_FEATURE_LEVEL_11_1) {
  enabled.logicOp                               = VK_TRUE;
  enabled.vertexPipelineStoresAndAtomics        = VK_TRUE;
}

`

had to comment these lines (due to missing MoltenVK support): for getting up to D3D_FEATURE_LEVEL_9_1: enabled.sampleRateShading = VK_TRUE; enabled.shaderCullDistance = VK_TRUE; enabled.robustBufferAccess = VK_TRUE; for D3D_FEATURE_LEVEL_9_3: enabled.multiViewport = VK_TRUE; for D3D_FEATURE_LEVEL_10_1: enabled.fullDrawIndexUint32 = VK_TRUE; enabled.geometryShader = VK_TRUE; enabled.logicOp = supported.logicOp; for D3D_FEATURE_LEVEL_11_0: enabled.shaderFloat64 = supported.shaderFloat64; enabled.shaderInt64 = supported.shaderInt64; enabled.tessellationShader = VK_TRUE; enabled.shaderStorageImageWriteWithoutFormat = VK_TRUE;

with all these "hacks" it doesn't complain but crashes executing simple d3d11 apps because it may be using missing features..

of course MoltenVK supporting DXVK might be a herculean effort but DXVK has shown it's able to run lots of DX11 modern games and this will be awesome on Mac.. this is just a report of things needed to do (as a means of guiding priorities of things to implement)..

hope is useful for someone..

[johnothwolo]

How did you get wine to recognise moltenVK? I get

checking for -lMoltenVK... not found

[oscarbg]

I think had some issues but copied everywhre /lib /usr/lib /usr/local/lib..

[KenThomases]

You configure with LDFLAGS="-L/path/to/the/MVK/libdir". At runtime, you need to set the DYLD_FALLBACK_LIBRARY_PATH environment variable to contain the same path. However, System Integrity Protection will strip any DYLD_* environment variables when a system executable is run and that included shell script interpreters. So, depending on how you're running Wine, you may need to take extra measures.

If you're running your own build using the top-level "wine" script, create a file names .winewrapper next to it that contains "export DYLD_FALLBACK_LIBRARY_PATH=/path/to/the/MVK/libdir:/usr/lib". That will be automatically sourced by the wine script at a point after SIP has done its stripping.

If you're running an installed Wine, then the "wine" command may be just the Wine loader program, not a shell script. In that case, SIP is not involved.

[iain-benson]

I've been experimenting with this on my fork (https://github.com/iain-benson/MoltenVK/tree/dx11), as I was hoping to play Witcher 3 on the Mac, without rebooting.

• Added (and implemented) vkGetPhysicalDeviceFeatures2KHR and vkGetPhysicalDeviceProperties2KHR
• Added stubs for the remaining methods for VK_KHR_GET_PHYSICAL_DEVICE_PROPERTIES_2, and the methods listed above
• Set the flags to true for the device features required for DX11

It gets further, and starts setting up the shader modules, but the conversion of the SPIR-V code fails as it tries to use an undefined method spvTexelBufferCoord.

I am fully aware the changes I made wouldn't actually result in a working system, but I was looking to see what work was required, and will continue attempting to implement what I can, to see if I can get anything working.

[oscarbg]

Hi @iain-benson, nice work.. strange thing is spvTexelBufferCoord seems to be introduced recently in https://github.com/KhronosGroup/SPIRV-Cross/blob/master/spirv_msl.cpp by : https://github.com/KhronosGroup/SPIRV-Cross/commit/4c5142b9d378ee3885ab6e49070d745f7e1e9318 "Emit and use spvTexelBufferCoord() function to convert 1D texel buffer coordinates to 2D Metal texture coordinates."

uint2 spvTexelBufferCoord(uint tc)
{
    return uint2(tc % 4096, tc / 4096);
}

just for curiosity what MoltenVK version are you using? from 1.1.82?

[cdavis5e]

So far, of those extensions, I've implemented VK_KHR_maintenance1, VK_KHR_shader_draw_parameters, VK_KHR_get_physical_device_properties2, and VK_KHR_descriptor_update_template. The patches for the first three were merged (#230, #232, #235). The patch for the last one depends on another patch I have to implement VK_KHR_push_descriptor (#236), which vkd3d (a different library for D3D12) wants but doesn't need to work. The reason for that is that those two extensions interact--there's a call that's available only if both extensions are supported. You can see the patch for VK_KHR_descriptor_update_template at my fork.

I still need to investigate the core Vulkan features that DXVK needs but MoltenVK doesn't have yet.

[oscarbg]

Amazing work @cdavis5e !! I thought we were months from seeing real progress on this.. Thanks..

[banzr]

Hey, I'd love to help but I'm not sure where to start. Can someone help fill me in on what references/documentation are out there? Thanks! I stumbled on this while trying to install Steam Proton on my macbook pro. I'm a Sr CS undergrad with no experience with DXVK or vulkan, but I am very familiar with C/C++.

[oscarbg]

Hi @banzr, I don't know what's the best answer I can gave you, but I will try.. basically for this specific issue you need to gain some Metal and Vulkan knowledge/experience.. as this is what MoltenVK is about.. For Metal see: https://developer.apple.com/documentation/metal (deprecated) https://developer.apple.com/library/archive/documentation/Miscellaneous/Conceptual/MetalProgrammingGuide/Introduction/Introduction.html useful will be: https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf also see : https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf (some things are only avaiable in newer OS or GPUs) then familiarize with MoltenVK codebase and also SPIR-V Cross as this is used for shader translation.. then seeing the extensions/optional Vulkan features DXVK needs (first post in this thread) and try to know how you map these to Metal and implement it.. in theory, you don't even need to know anything about DXVK.. as far as MoltenVK exposes all required DXVK Vulkan functionality you are done.. or you may forgot everything I said until now and go the opposite way.. of learning as needed to implement individual items.. let's take two examples: 1)Imagine you want to add double precision support to MoltenVK (really DXVK can use it but not useful in real games): you read Metal-Shading-Language-Specification and see: "Metal does not support the double.." so it can't be done.. well not easily.. it can be emulated by doing something similar to guys in Mesa have done: "The Student Working On "Soft" FP64 Support Is Good News For Older GPUs".. on mesa cgit you should find shader code emulating doubles via floats.. 2) Similar thing will be found for geometry shader, you will find Metal doesn't support it.. 3) Imagine you want to add tesselation support to MoltenVK: read in Apple how it's exposed: https://developer.apple.com/library/archive/documentation/Miscellaneous/Conceptual/MetalProgrammingGuide/Tessellation/Tessellation.html find a Vulkan simple sample: https://github.com/SaschaWillems/Vulkan/tree/master/examples/tessellation you will find MoltenVK uses SPIR-V Cross for shader translation you will need to add translation from tesselation stage to Metal..

hope it helps (a little)..

[cdavis5e]

SPIRV-Cross #675 should fix the issue with spvTexelBufferCoord().

[oscarbg]

@cdavis5e I see you are contributing lots of SPIRV-Cross patches/improvements too.. thanks.. let me ask two questions:

1. are they fixing (simiar to spvTexelBufferCoord issue) some MoltenVK bugs for DXVK or VKD3D? I say as sadly seems MoltenVK is still referencing an "old" SPIRVCross tag without all you improvements so would l like to know what benefits I get by compiling SPIRV-Cross up to latest commit..
2. will all your MoltenVK & SPIRV-Cross commits (& possibly your pull requests not yet merged on this two repos) is DXVK simple included apps like d3d11-triangle and d3d11-compute working? have you an estimate of how much work/patches still remaining to get these two "simple" demos (d3d11-triangle and d3d11-compute) working under MoltenVK? thanks..

[billhollings]

@oscarbg I'll be doing a MoltenVK release this weekend that will include an upgrade to the latest SPIRV-Cross...including all the wonderful work @cdavis5e has been contributing.

[cdavis5e]

@oscarbg Yes, these are to fix bugs in MoltenVK for vkd3d (which will inevitably benefit DXVK as well). Eventually, I would like to get the DXVK samples running. Hopefully I should be able to have them up within about a week—a month, tops.

Real support for cull distances (which DXVK apparently wants to create a device at all) might be difficult, though: Metal doesn't support them directly, so we'd probably need to have some sort of pre-processing stage to support that at all. In the meantime, I wonder if we could get away with saying we have it even though we don't. Logic op is similar, though I'm less sanguine about the prospects for emulating that one.

The big ones, of course, are geometry and tessellation shaders. Since Metal doesn't directly support tessellation control (hull) or geometry shaders, those need preprocessing similar to cull distances. My plan right now is to run these shaders as part of a compute pipeline that runs right before the graphics pipeline. This is the way Apple actually recommends tessellation be done. I think we can fully support geometry shaders this way, too. Supporting the two together will be a real pain, though, since we'll need to run three preprocessing steps before we can run the graphics pipeline: the vertex and tess. control shader in a compute kernel, the tessellator and tess. evaluation (domain) shader, then the geometry shader in another compute kernel.

I think it should be possible to just turn on robustBufferAccess, shaderImageGatherExtended, and drawIndirectFirstInstance right now. Most of the remaining needed extensions and features shouldn't be too hard to implement.

[KenThomases]

This may not be a good idea, but it's possible to do geometry shading in a Metal vertex shader if you change the draw command. See my StackOverflow answer here. This is mostly useful if rasterization is enabled, since it can feed directly into the rasterizer stage without switching from render to compute and back to render.

My SO answer was mostly contemplating hand translation. For automated translation, you'd generate code that parallels the geometry shader much more closely (because you pretty much have to). The "emit" operations would, first, track which output primitive and vertex within that primitive they're working on. Then, they'd check if the current output primitive and vertex are the ones for the current thread/invocation. If so, they'd set the output value. If not, they'd do nothing.

[oscarbg]

@billhollings many thanks for updating SPIRV-Cross and doing a release!

@cdavis5e also many thanks for your great detailed overview.. as you talk about cull distances & "pre-processing stage to support that at all".. I would love to know what do you think also about swizzle Metal limitations.. seems will require similar preprocess stage and seem like AAA Vulkan games (like Doom and Wolfenstein II) and other Vulkan programs (RPCS3 via Wine ( https://github.com/KhronosGroup/MoltenVK/issues/205 ) ) make use of it.. I posted results of running these two games under Wine+MoltenVK around a month ago here showing swizzle limitations: https://github.com/KhronosGroup/MoltenVK/issues/204#issuecomment-410453254 https://github.com/KhronosGroup/MoltenVK/issues/204#issuecomment-410453555

really I was just asking about d3d11-triangle and d3d11-compute DXVK samples as seems DXVK is asking for a lot of things, that these simple two programs don't need to run, and even simple hacking DXVK to not use missing caps&exts. , as it's shown in first post, samples still crash.. last time I tested they were failing at spvTexelBufferCoord step.. so maybe they are fixed now or may work soon as you share are also interested in getting it working..

@KenThomases thanks for sharing the idea here.. maybe that's what's doing Vmware in his Fusion product.. as you know probably for almost a year since Fusion 10 it supports DirectX10 via Metal.. I have run some DX10 geometry sample programs and it works OK.. mainly from NV DX10 SDK(http://developer.download.nvidia.com/SDK/10/direct3d/samples.html): Lightning,Rain,MetaBalls ,Fur - Shells and Fins.. also interesting is "Lightning" sample uses also stream out: "efficiently implemented using geometry shaders and stream out" so Vmware is emulating stream out also in Metal, I say because stream output much in the news these days like on khronos.org "Some Vulkan members are developing a multi-vendor EXT extension for transform feedback" right now because seems is the last big Vulkan "missing feature" for DXVK playing DX11 games using it "correctly" like "Witcher 3".. so eventually might make sense for MoltenVK to support also that extension..

[cdavis5e]

@oscarbg I just hit an issue that will require swizzling (related to #243). In this case, D3D puts the stencil value in green, but Metal puts it in red. By default, Vulkan also puts it in red; that's why vkd3d uses a swizzle to make it appear in green. To make this work for any arbitrary swizzle, we'll probably need to apply a technique similar to what wined3d does when the GL_ARB_texture_swizzle extension isn't available, by manually swizzling the texture and/or fixing up the shader to do a swizzle on every load.

[cdavis5e]

OK, I've implemented support for all extensions that DXVK needs, and I have some patches outstanding to implement the remaining few it wants in addition (#273), and I've also turned on and/or implemented quite a few core features as well. If I then force on all the other features that DXVK needs (see attached patch), I can get MS's D3D11 samples to successfully create a device using DXVK... and then it crashes and burns because DXVK sets a (0, 0, 0, 0) swizzle and Metal doesn't support arbitrary swizzles. Don't worry; I'm working on the swizzling issue: see https://github.com/KhronosGroup/SPIRV-Cross/pull/706.

[doitsujin]

I could get rid of the (0,0,0,0) swizzle if that helps. It's only used to provide support for unbound textures, but since the image used with that view is cleared to zero anyway, i could just use an identity swizzle there.

Some other swizzles are necessary, though.

[Rastafabisch]

I just wanted to report the current status to those curious (like I've been). With the latest MoltenVK as of today (v1.0.30 - including @cdavis5e preliminary DXVK patch), the latest non-rc wine version (3.21) and the latest DXVK libraries (0.94) I was able to launch the Witcher 3. While it does not crash, and starts playing the prolog audio and reacts to keyboard input (skipping the prolog), it does not draw anything to the screen. It's just black (using DXVK 0.81 - the last pre stream output version the screen was blue but apart from this it was just about the same. So to draw a line: Anyone involved already achieved some huge progress.

About the only error messages regarding dxvk/moltenvk are the following:

[***MoltenVK ERROR***] VK_ERROR_INITIALIZATION_FAILED: Vertex shader function could not be compiled into pipeline. See previous error.
err:   DxvkGraphicsPipeline: Failed to compile pipeline
err:     vs  : VS_71ac5766523bfe85650fb70fe2faa3b7a4efa3b8
err:     fs  : FS_a06e822f5ba921672573f4b95e59a40b14bf6ed7
[***MoltenVK ERROR***] VK_ERROR_INITIALIZATION_FAILED: Shader library compile failed (error code 3):
Compilation failed: 

program_source:8:31: error: 'function_constant' attribute requires Metal language standard macos-metal1.2 or higher
constant bool cb0_bound_tmp [[function_constant(0)]];

Here is the entire wine log, in case it proves to be somewhat helpful: Wine_MoltenVK_DXVK.txt

Wine was actually build with an older version of MoltenVK. Just the runtime library have been updated, though I do not expect it to be a problem.

[billhollings]

@Rastafabisch

What version of macOS are you running?

The complaint about requiring MSL 1.2 or higher would imply you are running macOS 10.11.

[cdavis5e]

The log also says that MSL 2.1 is supported, as well as GPU Family 2 and GPU Family 1 v4, implying that this was run on macOS 10.14.

[cdavis5e]

It looks like I need to implement geometry shader emulation and VK_EXT_transform_feedback to make this particular game work:

err:   D3D11: CreateGeometryShaderWithStreamOutput:
  Transform feedback not supoorted by device

[cdavis5e]

Here's my list of things still to be done for DXVK:

• [ ] Geometry shader emulation
• [ ] Cull distance emulation
• [x] Tessellation shaders
• [ ] VK_EXT_transform_feedback

Without cull distances, DXVK won't even let you create a 10level9 (D3D_FEATURE_LEVEL_9_1) device. My preliminary patch right now fakes the first three, which allows device creation to succeed. But if you attempt to use the unimplemented features...

[Rastafabisch]

@Rastafabisch

What version of macOS are you running?

The complaint about requiring MSL 1.2 or higher would imply you are running macOS 10.11.

The log also says that MSL 2.1 is supported, as well as GPU Family 2 and GPU Family 1 v4, implying that this was run on macOS 10.14.

Indeed I‘m running the latest stable version of Mojave. Having my build environment set up now, I can do some additional testing for new patches if needed.

[cdavis5e]

I wonder if this is a bug in Mojave--or even worse, intended behavior; i.e. it's looking at the version the binary was built for and using that to set the version of MSL to use. In either case, I think we can work around this by explicitly setting MTLCompileOptions.languageVersion to the latest version supported by the host system.

[Rastafabisch]

How would I actually insert this option? And do you think, it would already viable or shall I want until some additional patches and features are available/implemented?

[cdavis5e]

Try applying this patch to MoltenVK.

[billhollings]

Hmmm...the doc for MTLCompileOptions.languageVersion explicitly states:

By default, the most recent language version available is used.

So...it's worth a shot...but I'll be surprised (happily so...but silently cursing Apple) if your patch changes anything.

[Rastafabisch]

I just tested this patch after pulling the current 1.0.31 build. As the patch would need to be rebased I applied it manually instead and it compiled fine. However I do not get the improved swizzling support enabled during build.

Even if mslOpts.swizzle_texture_samples = context.options.shouldSwizzleTextureSamples; is reversed to mslOpts.swizzle_texture_samples = true; the wine.log is spammed with [***MoltenVK ERROR***] Pipeline does not support component swizzle errors.

The difference compared to the previous trial so war is, that the black screen got replaced by an pale violet one, and the game apparently freezes bevor I get audio. However now I get a themed mouse cursor.

I'll attache the current wine.log anyway. Wine_MoltenVK_DXVK.txt

[cdavis5e]

Try running it with the MVK_CONFIG_FULL_IMAGE_VIEW_SWIZZLE environment variable set to 1.