Allow creating process with injected DLL

[chyyran]

I am trying to hook Vulkan in an application, and due to how the Vulkan loader works, it doesn't seem possible to hook after the application calls vkCreateInstance.

[X64.Function(X64.CallingConventions.Microsoft)]
[X86.Function(X86.CallingConventions.Stdcall)]
[UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
public unsafe delegate Result vkQueuePresentKHR(Queue queue, PresentInfoKHR* pPresentInfo);```

...

var handle = Kernel32.GetModuleHandle("vulkan-1");

// QueuePresentKHRFn is never called.
this.VkQueuePresentKHRHook  = ReloadedHooks.Instance.CreateHook<vkQueuePresentKHR>(QueuePresentKHRFn, (long)handle.GetProcAddress("vkQueuePresentKHR")).Activate();

I realize that the canonical way of doing Vulkan hooking is via layers, but doing it via a validation layer complicates .NET interop, so if possible it would be nice to do it the 'traditional' way.

To be able to hook Vulkan properly, I need to be able to obtain a handle to the created VkInstance, and thereafter I can use vkGetInstanceProcAddr. vkCreateInstance is called directly via vulkan-1.dll exports, so if I can hook the process before the target application creates its instance, I can obtain a handle to the VkInstance and resolve the other functions via vkInstanceGetProcAddr.

Detours provides DetourCreateProcessWithDllEx which does injection before the process is fully loaded, so I can hook vkCreateInstance as early as possible. I was wondering if this functionality could also be provided in Reloaded.Injector.

Alternatively maybe I'm doing things completely wrong and there's a much easier way to hook Vulkan calls without needing early-hooking, in which case I would be happy to do instead.

[Sewer56]

A few ways to approach this.

One that doesn't involve modifying the target folder whatsoever is as follows:

• Launch the Process Suspended (CreateProcessW + CREATE_SUSPENDED).
• Inject your DLLs.
• Resume the primary thread to start the process. (procInfo.hThread).

Only problem is this approach deadlocks during one of the initialisation steps with this specific library, so I wound up creating a simpler injector for another project that needed this sort of behaviour.

Was something related to grabbing the address of kernel32 IIRC.

[chyyran]

I tried creating the process suspended and indeed ran into the deadlocking issues mentioned.

Rather than having to create another injector library, for my use case specifically I can probably 'inject' my CLR loader as a Vulkan layer and just pass in pfnNextGetInstanceProcAddr to the entry point, then do hooking as usual. Theoretically I think that should work though I haven't tried it out yet. Just a bit disappointing that I can't do it the same way as the other graphics APIs.