iOS installation - OK but VK_ERROR_OUT_OF_DEVICE_MEMORY

[AlexMorgand]

Hey! First of all thank you for your work.

I managed to get the app installed but I'm not sure if I did things correctly. Here is a walkthrough of what I did.

Initially did

mkdir build
cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build
cmake --build ./build -j4

I had to fix a couple of things first:

1. Remove case VK_DRIVER_ID_MESA_AGXV in vk_enum_string_helper.h
2. Replace define VULKAN_HPP_TYPESAFE_CONVERSION to VULKAN_HPP_TYPESAFE_CONVERSION 0 as it caused issues as it's expected to be an integer

Installed Vulkan SDK and prebuilt MoltenVK libraries.

and then running

make xcode_setup

in the build folder. I was expecting a xcodeproj to be generated but I got nothing.

What worked for me:

1. building in the root dir and not the build folder.
2. Using the default "apps/apple/VulkanSplatting/project.xcconfig" project
3. In MetalWindow.cpp, replaced context->instance->createMetalSurfacEXT(surfaceCreateInfo) to static_cast<VkSurfaceKHR>(context->instance->createMetalSurfacEXT(surfaceCreateInfo))
4. Removed the default VulkanSplatting.cpp (xcode generated) since it should be Renderer.cpp (I think?)
5. Fix references for glm, 3dgs.hpp and placed 3dgs.cpp correctly in the project since there was undefined references.

The app seems to be installed well but when loading a classic ply like bonsai it loads but when I move around to explore the splat, I get a VK_ERROR_OUT_OF_DEVICE_MEMORY

My device is a Iphone 15 max pro and the splats tested are not super big.

Any advice? Is there any more spot left in the testflight :)

Thanks in advance!

[shg8]

Thanks for the detailed writeup. The iOS wrapper isn't included in the CI/CD pipeline, and I haven't been keen on updating it. Without this entitlement, I believe the memory limit for iOS apps is around 50% to 60% of your total DRAM. Rendering a splat usually requires much more memory than the splat itself. The renderer isn't optimized for low memory either, as it requests double the amount of memory every time it runs out of memory. You can also trade efficiency for reduced memory usage by rasterizing tiles sequentially.

Unfortunately, I don't have plans to optimize this for mobile devices in the near future. If this is important for your use case and you'd like to work on it, I'd be happy to provide a few pointers on optimization opportunities.

[AlexMorgand]

Hey @shg8, thank you for your answer!

That's a shame as I've a hard time finding a GS viewer matching your quality :) Seems like iOS 3DGS rendering is a tough one. Was your demo on testflight really laggy?

You can also trade efficiency for reduced memory usage by rasterizing tiles sequentially I'll give a try. How we can get 60fps out of it :)

If this is important for your use case and you'd like to work on it, I'd be happy to provide a few pointers on optimization opportunities. If you can that would be terrific :)

Anything remarks on how I made the app work? Did I do something fundamentally wrong

[shg8]

Hey @shg8, thank you for your answer!

That's a shame as I've a hard time finding a GS viewer matching your quality :) Seems like iOS 3DGS rendering is a tough one. Was your demo on testflight really laggy?

You can also trade efficiency for reduced memory usage by rasterizing tiles sequentially I'll give a try. How we can get 60fps out of it :)

If this is important for your use case and you'd like to work on it, I'd be happy to provide a few pointers on optimization opportunities. If you can that would be terrific :)

Anything remarks on how I made the app work? Did I do something fundamentally wrong

Hi Alex, sorry for the delayed response. I think rendering tiles sequentially would save memory but dramatically increase rendering latency.

As for optimization opportunities, the first item would be to write a proper parallel radix sort. I believe someone has already done this. Also, check out his implementation.

[0xMihir]

I'm getting this on an M1 Pro too:

[mvk-error] VK_ERROR_OUT_OF_DEVICE_MEMORY: Lost VkDevice after MTLCommandBuffer "vkQueueSubmit MTLCommandBuffer on Queue 0-0" execution failed (code 3): Caused GPU Address Fault Error (0000000b:kIOGPUCommandBufferCallbackErrorPageFault)

This is right after it tries to reallocate buffers for sorting