Detouring Windows API GetDeviceCaps Function Prevents Programs From Starting

[Stonyx]

I have the following basic DLL code that detours the Windows API WindowsEnumDisplaySettingsA function:

static BOOL(WINAPI* WindowsEnumDisplaySettingsA)(LPCSTR lpszDeviceName, DWORD iModeNum,
  DEVMODEA* lpDevMode) = EnumDisplaySettingsA;

BOOL WINAPI DetouredEnumDisplaySettingsA(LPCSTR lpszDeviceName, DWORD iModeNum,
  DEVMODEA* lpDevMode)
{
  // Call the real GetDeviceCaps function
  return WindowsEnumDisplaySettingsA(lpszDeviceName, iModeNum, lpDevMode);
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
  // Restore the in memory import table after we are loaded by withdll.exe
  DetourRestoreAfterWith();

  // Switch based on the reason for this function call
  switch (ul_reason_for_call)
  {
  case DLL_PROCESS_ATTACH:

    // Attach the function detours
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(&(PVOID&)WindowsEnumDisplaySettingsA, DetouredEnumDisplaySettingsA);
    DetourTransactionCommit();

    break;
  case DLL_PROCESS_DETACH:
    // Detach the function detours
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourDetach(&(PVOID&)WindowsEnumDisplaySettingsA, DetouredEnumDisplaySettingsA);
    DetourTransactionCommit();

    break;
  }

  return TRUE;
}

which works great when starting the program with the following command:

withdll.exe /d:test.dll program.exe

However, if I change the function being detoured to GetDeviceCaps and end up with the following code (the only changes are the function name and signature):

static int (WINAPI* WindowsGetDeviceCaps)(HDC hdc, int index) = GetDeviceCaps;

int DetouredGetDeviceCaps(HDC hdc, int index)
{
  // Call the real GetDeviceCaps function
  return WindowsGetDeviceCaps(hdc, index);
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
  // Restore the in memory import table after we are loaded by withdll.exe
  DetourRestoreAfterWith();

  // Switch based on the reason for this function call
  switch (ul_reason_for_call)
  {
  case DLL_PROCESS_ATTACH:

    // Attach the function detours
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(&(PVOID&)WindowsGetDeviceCaps, DetouredGetDeviceCaps);
    DetourTransactionCommit();

    break;
  case DLL_PROCESS_DETACH:
    // Detach the function detours
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourDetach(&(PVOID&)WindowsGetDeviceCaps, DetouredGetDeviceCaps);
    DetourTransactionCommit();

    break;
  }

  return TRUE;
}

and start the application with the exact same command, it simply fails silently without any errors displayed. In the Event Viewer I find the following error:

Faulting application name: program.exe, version: 0.0.0.0, time stamp: 0x5432e000
Faulting module name: TEST.dll, version: 0.0.0.0, time stamp: 0x66ff615a
Exception code: 0xc0000005
Fault offset: 0x62010b0b
Faulting process id: 0x15c0
Faulting application start time: 0x01db1611a56f2db4
Faulting application path: C:\Path\To\Program\program.exe
Faulting module path: C:\Path\To\Program\TEST.dll
Report Id: 79a05641-2f2a-48d3-b2b2-ae9b96d7f60f
Faulting package full name: 
Faulting package-relative application ID: 

The exception code seems to stand for STATUS_ACCESS_VIOLATION according to https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-erref/596a1078-e883-4972-9bbc-49e60bebca55 which must be happening somewhere in the Detours code because nothing in the DLL code should be throwing this exception.

I've tried this with several different applications and they all fail silently when detouring the GetDeviceCaps function but work just fine if I detour a different function. Is there any other way to go about detouring the GetDeviceCaps function in an application without it preventing the application from starting?

[jedwardsol]

Try giving DetouredGetDeviceCaps the correct calling convention (WINAPI)

[BissetJ]

Agreed, looks like the wrong calling convention on DetouredGetDeviceCaps would have been the problem here.