Apple Silicon macOS port

[Anthony-Gaudino]

This is an initial Apple Silicon macOS port.

Application builds, but trying to run the editor fails due to Vulkan issues.

Metal doesn't support Geometry shaders which are used in WickedEngine , these seem to be the shaders:

for f in shaders/spirv/*.cso; do 
  spirv-dis "$f" -o - 2>/dev/null | grep -n "OpCapability Geometry" && echo " -> $f"
done
7:               OpCapability Geometry
 -> shaders/spirv/envMap_skyPS_dynamic.cso
7:               OpCapability Geometry
 -> shaders/spirv/envMap_skyPS_static.cso
7:               OpCapability Geometry
 -> shaders/spirv/envMapPS.cso
6:               OpCapability Geometry
 -> shaders/spirv/objectGS_voxelizer.cso
8:               OpCapability Geometry
 -> shaders/spirv/objectPS_prepass_alphatest.cso
8:               OpCapability Geometry
 -> shaders/spirv/objectPS_prepass.cso
6:               OpCapability Geometry
 -> shaders/spirv/voxelGS.cso

I tried simply ignoring geometry shaders without success, the editor runs without a crash, but it still point to errors, probably because of the unsupported geometry shaders.

There's a PR to implement geometry shaders in MoltenVK, so I also tried implementing it on latest stable forks: forking latest stable MoltenVK and also the necessary SPIRV-Cross

With this I got stuck on this crashing error while what seems to trying to render the first frame using Vulkan:

-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5843: failed assertion `Draw Errors Validation
Vertex Function(main0): missing buffer binding at index 30 for spvDynamicOffsets[0].

I built MoltenVK shared library and placed it on same dir as editor and have been using this command to run it to try to capture debug info:

MTL_DEBUG_LAYER=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation VK_LOADER_DEBUG=all VK_ICD_FILENAMES=./MoltenVK_icd.json DYLD_LIBRARY_PATH=./ ./Editor

I would probably need help to get this done.

---

Update

The editor runs stable, Sponza, Terrain with weather and the character controller scene works, but some tradeoffs were taken, unsolved bugs exists and some settings are unstable:

• The terrain works, but it's texturing uses a new path since MoltenVK doesn't support virtual textures. Some thin seams are visible between terrain chunks.

• There's a very annoying bug where black squares/stripes are visible on faces, the larger the face the higher the change that it happens. These back areas are not being processed trough a fragment shader.

• Voxel GI, SSGI and DDGI work, but they are not stable and might hang the process emitting errors like:

[Error] [CopyAllocator::submit] vkWaitForFences resulted in VK_TIMEOUT
[Error] [SubmitCommandLists] vkWaitForFences resulted in VK_TIMEOUT, fence statuses:
QUEUE_GRAPHICS = VK_NOT_READY
QUEUE_COMPUTE = VK_NOT_READY
QUEUE_COPY = VK_NOT_READY
QUEUE_VIDEO_DECODE = OK

Or just crash.

• Voxel GI uses a different path to bypass geometry shaders, it's quite hacky.

• There's a bug on the gizmo, while trying to move some element using the XY handle then the object coordinates are all set to nan. The same happens with the scale all axis handle.

• For some reason impostors visualized in a certain angle appear with incorrect colors (pink).

• Seems like IK and swimming are jittering.

Other issues might be present, I didn't test every single feature.

[turanszkij]

Looks pretty good, I skimmed through the changes. I can't test it on Apple, and I don't know what is causing the problems.

This is odd: -> shaders/spirv/objectPS_prepass.cso 6: OpCapability Geometry

It is a pixel shader which uses SV_PrimitiveID, if the primitiveID is not supported by hardware then we can emulate it with geometry shader, but you said that geometry shader is also not available so I'm not sure what could be done except using mesh shader. Is mesh shader supported?

[Snowapril]

Hello, I’m also interested in this project and working on porting it to the Metal API.

All M-series Macs support mesh shading, and on A-series devices it’s supported starting from the A14 (with some limitations before Apple9), according to the feature set table (https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf).

Have you considered trying the Metal Shader Converter (https://developer.apple.com/metal/shader-converter/)? It can convert traditional geometry and tessellation shader stages into object/mesh shader stages relatively easily.

[brakhane]

The DirectXShaderCompiler also seems to (now?) support compiling to Metal IR, might be worth taking a look.

[brakhane]

This is odd: -> shaders/spirv/objectPS_prepass.cso 6: OpCapability Geometry

According to the docs DXC declares Geometry capability when PrimitiveId is used as an input in a pixel shader.

[Anthony-Gaudino]

Have you considered trying the Metal Shader Converter (https://developer.apple.com/metal/shader-converter/)? It can convert traditional geometry and tessellation shader stages into object/mesh shader stages relatively easily.

No, haven't tried that. Looks promising.

I was able to bypass the missing buffer binding at index 30 for spvDynamicOffsets[0] but then got into GPU page faults. The problem is probably within SPIRV-Cross or MoltenVK. If Metal Shader Converter can convert geometry and tessellation shader stages into object/mesh shaders then it could work.

---

The DirectXShaderCompiler also seems to (now?) support compiling to Metal IR, might be worth taking a look.

Will investigate this too.

[Snowapril]

The DirectXShaderCompiler also seems to (now?) support compiling to Metal IR, might be worth taking a look.

I didn’t know that DXC supports Metal IR generation, but it seems to rely on the Metal Shader Converter and is still missing several features (such as compatibility flags, ray tracing pipelines, etc.). The Metal IR generation pipeline using the Metal Shader Converter is quite straightforward — it involves generating DXIL with DXC and then converting it through the Metal Shader Converter, which is essentially the same process as DXC’s internal Metal generation flow.

Will investigate this too.

@Anthony-Gaudino I see your work now, and it looks like you’re aiming to use MoltenVK to leverage existing Vulkan implementations. If you decide to use the Metal Shader Converter instead, you’ll need to implement your own native Metal API code.

[Anthony-Gaudino]

@Anthony-Gaudino I see your work now, and it looks like you’re aiming to use MoltenVK to leverage existing Vulkan implementations. If you decide to use the Metal Shader Converter instead, you’ll need to implement your own native Metal API code.

Yes, I used MoltenVK because I though it would be easier and require less changes.

I looked into the Metal shader converter and for handling Geometry and Tesellation pipelines I would need to use the Metal shader converter companion header. For this I would need to either implement this in MoltenKV or on a Native Metal API as you said.

Another option is to continue on forked MoltenVK path and try to make it work.

And the third option is to use the stable MoltenVK and change existing shaders so that they don't depend on Geometry pipeline or Tesselation; or have separate shaders or code paths for Metal.

I created another branch that follows option 3. In this new branch I was able to run the Editor and basic features work, complex actions, like trying to add a terrain still crash.

[turanszkij]

I implemented a workaround for SV_PrimitiveID, which will be useful for this pull request, now the prepass object rendering shaders (objectPS_prepass.cso) don't use the SV_PrimitiveID, so the problem will be gone for those.

[turanszkij]

The commit above removes SV_RenderTargetArrayIndex from envmap pixel shaders, so they are not marked with Geometry shader capability.

[turanszkij]

The objectGS_voxelizer is a proper geometry shader, that will just have to be not used on Apple, but it will be kept for other platforms (it is optional, not required usually, only used for voxel gi)

[Anthony-Gaudino]

The commit above removes SV_RenderTargetArrayIndex from envmap pixel shaders, so they are not marked with Geometry shader capability.

Thank you! I will merge those changes!

[Anthony-Gaudino]

HI @turanszkij

Sky PS shaders are still reported as using GS:

for f in shaders/spirv/*.cso; do                                                    
  spirv-dis "$f" -o - 2>/dev/null | grep -n "OpCapability Geometry" && echo " -> $f"
done
7:               OpCapability Geometry
 -> shaders/spirv/envMap_skyPS_dynamic.cso
7:               OpCapability Geometry
 -> shaders/spirv/envMap_skyPS_static.cso
6:               OpCapability Geometry
 -> shaders/spirv/objectGS_voxelizer.cso
6:               OpCapability Geometry
 -> shaders/spirv/voxelGS.cso

[turanszkij]

@Anthony-Gaudino You're right, now it should be fixed.

[Anthony-Gaudino]

@turanszkij,

Currently on macOS, textures are being rendered with greyscale colors. I would like to confirm with you if Base Color is really supposed to be BC1_UNORM.

[turanszkij]

BC1 is correct, it supports RGB colors. You can disable the compression to have regular RGBA format for the material by this checkbox, and also check that saturation slider is at default (1) because that can be used to make colors greyscale if it's at 0.

[Anthony-Gaudino]

BC1 is correct, it supports RGB colors. You can disable the compression to have regular RGBA format for the material by this checkbox, and also check that saturation slider is at default (1) because that can be used to make colors greyscale if it's at 0.

@turanszkij

Have looked into those, but it must be something different. Would you have any other clue of what could cause this issue?

[turanszkij]

You could take a look at that correct swizzle is used for textures (no idea how to look at it on Apple though). By the way if you set just material color that shows up correctly?

[Anthony-Gaudino]

You could take a look at that correct swizzle is used for textures (no idea how to look at it on Apple though). By the way if you set just material color that shows up correctly?

Yes, it does. If I set the color using the color picker it displays the correct color.

[Anthony-Gaudino]

Hi @turanszkij,

I’ve updated the branch with the latest changes and also updated the PR comment. There are still a few unresolved issues that I haven’t been able to fix, and I won’t be able to continue working on the macOS port for now. Hopefully, the current progress will be useful for anyone who wants to continue from here.

Someone with more graphics programming experience, like @Snowapril, might be able to make good progress on it if interested.

[turanszkij]

Thanks, it would be nice if the compile errors were resolved, after that I consider merging it to not lose all progress.

[Snowapril]

@Anthony-Gaudino Thanks for sharing! I'm also interested in getting Wicked Engine running on macOS, so providing fixes for various compile errors with the Apple Clang compiler would be really helpful.